SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
SAFIRE, an 83K-image fire and smoke benchmark, shows open-source multimodal LLMs average only 61.9% accuracy on safety-critical fire reasoning.
SAFIRE is a large-scale benchmark for fire-smoke understanding in multimodal LLMs with 83K captioned images across 20 scenarios and 193K multiple-choice VQA questions spanning 10 evaluation dimensions from perception to higher-order reasoning. Annotations were built via a GPT-5.4-assisted multi-stage pipeline with MLLM majority voting. Ten open-source MLLMs (8B-38B) average 61.9% accuracy, exposing major gaps in safety-critical reasoning. Adapting vision encoders on 7% of the domain data raises fire-scene classification from 20.1% to 64.5%.
- 83K captioned images across 20 scenarios; 193K MCVQA over 10 evaluation dimensions
- Ten open-source MLLMs (8B-38B) average 61.9%, revealing safety-critical reasoning gaps
- Domain adaptation with 7% of the data lifts fire-scene classification from 20.1% to 64.5%
- GPT-5.4-assisted pipeline plus MLLM majority voting ensures annotation quality
From the original article
1 min read at arxiv.orgMultimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation.
By Pengfei Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.