EvaOmni scores Omni LLMs on evidence-grounded audiovisual reasoning
Original title: EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning
Gemini3.7-Flash led 24 models at 79.4% accuracy, but Level 3 bottleneck grounding fell to 12.3%.
Researchers released EvaOmni, with 392 videos and 1,027 human-annotated question-answer pairs spanning 15 capabilities and three levels. Models must return an answer and supporting evidence intervals so scoring covers the reasoning chain. Coverage and Bottleneck Grounded Scores are folded into answer correctness. Gemini3.7-Flash led overall, while complex-reasoning localization remained weak.
- Experts mark audio-visual evidence and build video distractors before human review.
- Models later filter textual and unimodal shortcuts from the questions.
- The top model scored 70.8% coverage grounding and 35.1% bottleneck grounding.
- At Level 3, accuracy was 79.1% while bottleneck grounding was 12.3%.
- Products
- EvaOmni
- AI models
- Gemini3.7-Flash
From the original article
1 min read at huggingface.co
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer.
By Xiaoxing Hu, Wenxi Yao, Chaoyuan Shi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.