Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Prism uses dynamic sparse attention to train joint 2K video-audio models 2.5× faster than full attention.
Prism is a dynamic sparse-attention method for natively training joint video-audio generators at 2K resolution. It groups tokens into spatiotemporal macro-zones and uses channel-wise video feature variance plus audio-to-video cross-attention norms to choose finer blocks where visual change or audio-visual coupling is strong. A hybrid block-selection strategy sets per-query sparsity. Experiments report a 2.5× training speedup versus full attention and higher generation quality.
- Designed for native 2K joint video-audio training
- Block shape follows visual variance and audio-visual coupling
- Hybrid selection sets sparsity per query
- Reports 2.5× speedup and better quality than full attention
From the original article
1 min read at huggingface.co
Natively training joint video-audio generation models at higher resolutions empowers them to learn richer visual details and sharper motion dynamics. However, full attention incurs quadratic cost and, as resolution increases, spreads attention over increasingly redundant tokens, diluting learning signals for informative content and disrupting pretrained priors.
By Shuyuan Tu, Qi Tian, Yinming Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.