Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
LLM crisis sitreps can match human completeness scores yet reproduce few official figures and omit provenance.
Researchers derive a 15-slot schema from professional humanitarian situation reports and use it to compare 121 human-authored reports with 261 LLM-generated reports across five disasters. On Cyclone Idai, Gemini-3.7-Flash scored 0.827 versus a human ceiling of 0.821, a difference statistically indistinguishable from zero (permutation p=.852) that did not replicate under GPT-5.6-Luna. Across three primary events, machine reports reproduced only 5.4–11.0% of same-day official figures, and report date, organisation, series, and authority never reached professional coverage. ROUGE-L was uninformative, and BERTScore was negatively associated with completeness after multiplicity correction.
- A 15-slot schema was derived from professional humanitarian situation reports.
- 121 human and 261 LLM reports were scored across five disasters and three hazard types.
- Gemini-3.7-Flash nominally beat the human ceiling on Cyclone Idai, but the gap was not significant.
- Machine reports reproduced only 5.4–11.0% of same-day official figures on the primary events.
- ROUGE-L was uninformative and BERTScore correlated negatively with completeness.
- AI models
- Gemini-3.7-FlashGPT-5.6-Luna
From the original article
1 min read at huggingface.co
Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.
By Sachin Kumar · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.