Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Video DeltaNet combines softmax and linear attention for video generation, cutting DiT denoising of a 14.3-second 768p video to 6.7 seconds.
Video DeltaNet (VDN) pairs local softmax attention with a bidirectional linear memory branch using Video Delta Attention, which updates memory once per frame while jointly incorporating its spatial tokens. It is instantiated on MiniMax H3, applying the hybrid to video-to-video interactions while retaining softmax attention for text or audio interactions. With eight-step distillation and an optimized SGLang serving stack, VDN-H3 completes denoising of a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs, a 14.5x speedup over the 50-step dense H3 baseline.
- Video Delta Attention updates linear memory once per frame with spatial tokens
- Staged teacher-alignment recipe progressively introduces the new pathway into pretrained models
- 14.5x speedup over the 50-step dense H3 baseline on eight B200 GPUs
- Retains softmax attention for interactions involving text or audio
- Tags
- diffusion-transformerhybrid-attentioninference-efficiencylinear-attentionminimaxvideo-generation
- AI models
- MiniMax H3
From the original article
1 min read at arxiv.orgVideo diffusion models repeatedly process long spatiotemporal token sequences during denoising, making attention a major computational bottleneck. Linear attention offers an appealing alternative and has been widely adopted in recent large language models, but directly applying it to video models often fails to preserve the fine-grained interactions required for high-quality generation.
By Haocheng Xi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.