EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment
EVA-Cap improves audiovisual captions by aligning them to atomic events with SFT and GRPO.
EVA-Cap treats audiovisual captioning as alignment to atomic events rather than unstructured text generation. It is trained with supervised fine-tuning and group relative policy optimization on EVA-Data, targeting global event completeness, intra-event attribution, and inter-event synchronization. The authors report about a 9.0% average gain on ChronusAV and a 3.3% gain on WorldSense versus existing methods.
- Captions are decomposed into atomic audiovisual events.
- Training uses SFT and GRPO on the EVA-Data set.
- GRPO targets completeness, attribution, and event synchronization.
- Reports +9.0% on ChronusAV and +3.3% on WorldSense.
From the original article
1 min read at huggingface.co
Recent advances in Omni Language Models (OLMs) provide a unified solution for audiovisual video captioning. Nevertheless, existing approaches predominantly treat captioning as an unstructured text generation process, neglecting the inherent semantic structure of videos and leading to suboptimal fine-grained audiovisual alignment.
By Jiapeng Shi, Qiuxia Hou, Jiaqi Leng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.