VideoLVR adds latent visual evidence to MLLM video reasoning
Original title: VideoLVR: Latent Visual Reasoning over Grounded Video Evidence
Latent tokens reconstructed from the input video let models revisit local frames without large context growth.
The paper says language chain-of-thought does not reliably let multimodal models revisit visual evidence, causing visual forgetting. VideoLVR builds latent visual evidence from local regions at relevant moments and supervises it with a visual-token prediction objective over a vocabulary derived from the input video. Sampled visual tokens can also be treated as actions and optimized with text using final-answer rewards. The authors report gains on video benchmarks, especially tasks that need fine-grained evidence.
Unconfirmed: The abstract does not report benchmark names or numeric gains.
- Textual coordinates name locations rather than visual content.
- Reinserting crops or frames preserves detail but expands context.
- Reinforcement learning jointly optimizes visual and text tokens.
From the original article
1 min read at arxiv.orgChain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual evidence, leading to visual forgetting and impairing their ability to reason over video content.
By Shilong Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.