Reviso prioritizes key visual evidence in video captioning
Original title: Reviso: Not All Visual Evidence Deserves Equal Attention in Video Captioning
Reviso builds CFP observation plans and trains with GRPO to outperform baselines on four benchmarks and improve downstream video question answering.
Multimodal large language models can overlook essential video context. Reviso learns observation strategies through hierarchical Context--Focus--Points plans and goal-directed active perception. It integrates verification-guided training with Group Relative Policy Optimization incorporating key-fact, Plan Alignment, and Unproductive Tool-Call rewards.
- Reviso constructs hierarchical CFP observation plans from low-frame-rate previews.
- It performs selective revisiting of temporal segments with tool integration.
- Training uses factual verification and teacher-guided refinement.
- Reviso outperforms open-source baselines on four video captioning benchmarks.
From the original article
1 min read at huggingface.co
Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video.
By Tengfei Liu, Yang Shi, Yuran Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.