The Attention Triangle in Audio-Video Models
Researchers analyze the 'attention triangle' in audio-video diffusion models, showing bias-driven cross-attention routing causes semantic leakage and proposing inference-time interventions that improve grounding.
A study probes the three cross-attention edges linking text, audio, and video streams in audio-video diffusion models. It finds the audio-video edge is bidirectional and shaped by parameter-encoded biases, so prompts in tension with learned priors can be overridden, producing visually canonical but incorrect outputs. Attention-derived signals are used as diagnostics and to guide inference-time interventions that improve cross-modal semantic grounding while preserving generation quality.
- Audio-video cross-attention routes semantics bidirectionally between sound and video streams
- Parameter-encoded biases can override prompt conditioning, producing canonical but incorrect outputs
- Attention-derived signals serve as diagnostics to deliberately incur and study leakage
- Inference-time interventions improve cross-modal semantic grounding without degrading generation quality
From the original article
1 min read at huggingface.co
Audio-video diffusion models rely on cross-modal attention to coordinate text, sound, and visual content, yet this same mechanism can introduce subtle and systematic semantic leakage.
By Sagi Polaczek, Noa Kraicer, Gal Metzer · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.