All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
RecCAR regularizes cross-attention so companion modalities better constrain joint video generation.
Joint multimodal diffusion transformers show a reciprocal correspondence gap: companion modalities attend strongly to video, but video is only weakly constrained in return. RecCAR is a KL regularizer that treats video-to-modality attention as a fixed reference and aligns the weaker modality-to-video direction toward it. Across joint video-motion and video-audio generation, the Human Anatomy score rises from 0.69 to 0.75 and audio-video desynchronization falls from 0.804 to 0.752.
- Reciprocal correspondence gap is measured over video tokens
- RecCAR applies KL regularization toward video-to-modality attention
- Human Anatomy score improves from 0.69 to 0.75
- Audio-video desynchronization falls from 0.804 to 0.752
Full article143 words · extracted from huggingface.co · click to collapse
Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event. We find that joint multimodal diffusion transformers exhibit a corresponding asymmetry in cross-modal correspondence: companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. We express both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap. We introduce RecCAR, standing for Reciprocal Cross-modal Attention Regularization, a KL regularizer that uses the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it. Across joint video-motion and video-audio generation, RecCAR improves the Human Anatomy score from 0.69 to 0.75 and reduces audio-video desynchronization from 0.804 to 0.752, while improving overall generation
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.27901