All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
RecCAR regularizes cross-attention so companion modalities better constrain joint video generation.
Joint multimodal diffusion transformers show a reciprocal correspondence gap: companion modalities attend strongly to video, but video is only weakly constrained in return. RecCAR is a KL regularizer that treats video-to-modality attention as a fixed reference and aligns the weaker modality-to-video direction toward it. Across joint video-motion and video-audio generation, the Human Anatomy score rises from 0.69 to 0.75 and audio-video desynchronization falls from 0.804 to 0.752.
- Reciprocal correspondence gap is measured over video tokens
- RecCAR applies KL regularization toward video-to-modality attention
- Human Anatomy score improves from 0.69 to 0.75
- Audio-video desynchronization falls from 0.804 to 0.752
From the original article
1 min read at huggingface.co
Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event.
By Ohad Rahamim, Dvir Samuel, Idan Schwartz · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.