titleLooking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
authorsSachin Kumar
date2026-10-13
venuearXiv (preprint, not peer reviewed)
idarXiv:2610.18762
upvotes3 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractEvaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95\% bootstrap CI [-.049,.059]; permutation p=.852), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen's κ=.799. All four judges clear the plan's raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected κ. Across the three primary events, machine reports reproduce 5.4-11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3-19.6% of official figures and 85.4-90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event-judge cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any cell. On the three primary events, ROUGE-L is uninformative, BERTScore is negatively associated with completeness after multiplicity correction, and a generic LLM judge changes sign.

cite

@misc{kumar2026looking,
  title={Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports},
  author={Sachin Kumar},
  year={2026},
  eprint={2610.18762},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.18762}
}