AVC-Reward evaluates audio-video consistency at the event level
Original title: AVC-Reward: Event-Centric Audio-Video Consistency Evaluation
Authors say the reward model aligns best with human judgments and can steer a joint generator via online reinforcement learning.
Researchers propose an event-centric framework for audio-video consistency as joint generators produce multi-event scenes, layered audio, and asymmetric cross-modal relations. AVC-Reward scores global narrative harmony, event-level timing, and fine-grained attribute alignment, and can localize failures. On AVC-Benchmark, built mainly from positive and negative samples of various generators, the authors report the strongest agreement with human judgment among automatic methods.
Unconfirmed: State-of-the-art human alignment and generation gains are author-reported, not independently verified.
- The metric has three levels: narrative, temporal sync, and attribute alignment.
- AVC-Benchmark uses positive and negative samples from multiple generators.
- Used as an online reinforcement-learning signal, it improved one joint model.
- The project page is tsuki0512.github.io/AVC-Reward.github.io.
- Products
- AVC-Benchmark
- AI models
- AVC-Reward
From the original article
1 min read at huggingface.co
As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks.
By Yuruo Wu, Kaihang Pan, Kai Shen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.