OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning
OmniCapBench diagnoses fine-grained audio-visual captioning errors across 786 annotated videos.
OmniCapBench reframes audio-visual caption evaluation as a structured diagnostic over atomic, verifiable units rather than free-form whole-caption scores. It covers three tracks—entity references, visual shots, and audio events—using deterministic constraint checks and localized LLM semantic comparisons on 786 densely annotated videos. The benchmark separates perception errors such as temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Frontier multimodal models show strong local perception but weak long-horizon audio-visual reasoning, especially on identity drift and cross-modal alignment.
- Evaluation uses atomic units for entities, visual shots, and audio events.
- Scoring mixes deterministic constraint checks with localized semantic comparisons.
- The set has 786 densely annotated videos.
- Frontier MLLMs show strong local perception but weak long-horizon audio-visual reasoning.
From the original article
1 min read at huggingface.co
Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits.
By Zhongyu Yang, Jiale Tao, Ruitao Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.