AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation
AV-GRPO applies modality-anchored decoupled diffusion RL to joint audio-video generation, beating LTX-2.3 on quality, alignment, and synchronization.
AV-GRPO is a modality-anchored online diffusion RL framework for joint audio-video generation, pairing three modules (modality-anchored rollouts, trajectory-locked frozen-tower optimization, adaptive objectives) with the 5DAV difficulty-controllable dataset. The approach decouples coupled multimodal preference learning into unimodal subproblems for better credit assignment and synchronization. On JavisBench and VABench, AV-GRPO outperforms LTX-2.3 in generation quality, semantic alignment, and cross-modal synchronization under both LoRA and full fine-tuning. Code and data are released on GitHub.
- Decouples multimodal preference learning into unimodal subproblems for reward attribution
- Frozen-tower optimization cuts compute cost of training two modality towers
- 5DAV dataset varies difficulty across five dimensions; beats LTX-2.3 on two benchmarks
From the original article
1 min read at huggingface.co
Recent years have witnessed major progress in joint audio-video generation. Existing models still suffer from limited per-modality fidelity, insufficient text-modality alignment and weak cross-modal synchronization. While reinforcement-learning post-training offers a promising remedy, directly adapting it to joint audio-video generation is challenging.
By Zhiyu Xu, Weilong Yan, Yufei Shi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.