SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation
SyncReward adds an 80,000-rating dataset and reward model for audio-visual synchronization.
The SyncReward paper studies human-aligned evaluation of audio-visual synchronization in generated speech and sound. The authors introduce SyncReward-Data-80K, with 80,000 professional ratings of clips from eight generators across Speech, Onset, Instrument, and Ambient events, plus SyncReward-Bench, a held-out set of 1,000 multiply rated clips. They report that existing synchronization metrics and omni-modal models disagree substantially with human judgments. SyncReward, trained with contrastive pretraining and human supervision, shows the strongest human agreement and improves generation through best-of-N selection and reinforcement-learning post-training.
- SyncReward-Data-80K contains 80,000 professional synchronization ratings.
- Ratings cover Speech, Onset, Instrument, and Ambient sound events from eight generators.
- SyncReward-Bench holds out 1,000 clips rated independently by multiple professionals.
- The reward model leads evaluated methods in agreement with human ratings.
- It improves sync via best-of-N selection and reinforcement-learning post-training.
- Organizations
- WeChatCV
- AI models
- SyncReward
From the original article
1 min read at huggingface.co
Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs.
By Kaisi Guan, Ying Ba, Wenjing Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.