LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception
LEAP retrieves short audio-visual windows for long recordings, beating Qwen3-Omni by up to 16.8%.
LEAP is a long audio-video question-answering method that splits a recording into fixed blocks, scores short candidate windows, and answers using only the highest-ranked windows. Peak context therefore stays independent of recording length, and the same grid supports streaming queries without streaming-specific training. Separate LoRAs train localization and answering. Across AVQA benchmarks, LEAP improves Qwen3-Omni-30B-A3B by 4.5-16.8% and MiniCPM-o 4.5 by 3.1-13.0% over its published results.
- LEAP scores short windows instead of encoding hour-scale recordings.
- Answer context stays bounded regardless of recording length.
- It beats Qwen3-Omni-30B-A3B by 4.5-16.8% on AVQA benchmarks.
- It also lifts MiniCPM-o 4.5 by 3.1-13.0% over published results.
- The block grid supports streaming without streaming-specific training.
- AI models
- Qwen3-Omni-30B-A3BMiniCPM-o 4.5
From the original article
1 min read at huggingface.co
Hour-scale audio-visual question answering is constrained by a context dilemma: dense whole-recording encoding rapidly exhausts context limits, whereas uniform temporal compression severely dilutes fine-grained acoustic and visual evidence. We introduce LEAP, a framework where the model retrieves its own evidence without placing the whole recording in one context.
By Juyi Lin, Zhiqiang Lao, Jiali Cui · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.