Nunchux AI Introduces VC-Attention: A Training-Free Low-Bit Attention Kernel That Speeds Up Video Diffusion Transformers
Nunchux AI introduces VC-Attention, a training-free low-bit attention kernel that speeds up video diffusion transformers up to 3.58x.
Nunchux AI unveiled VC-Attention, a training-free attention kernel for video Diffusion Transformers combining V-Smooth (k-means value-token grouping with block-mean residual quantization) and ExpCast-FP8 (single multiply-add softmax exponentiation). Benchmarks on Wan2.2-T2V-A14B, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3 show 1.59x attention speedup on B200 at 8-bit and 3.58x on RTX 5090 at 4-bit, with end-to-end gains up to 1.70x. It beats SageAttention2 by 2.3 dB PSNR on Wan2.2 at 8-bit and SageAttention3 by up to 3.6 dB at 4-bit. No public kernel release yet; a proprietary extension runs in Nunchux's stack.
- V-Smooth clusters value tokens and quantizes per-channel E4M3/NVFP4 residuals after block-mean subtraction
- ExpCast-FP8 replaces the FP32 softmax exponential and cast with one fused multiply-add
- Attention speedups: 1.59x on B200 (8-bit), 3.58x on RTX 5090 (4-bit) for Wan2.2
- Outperforms SageAttention2 and SageAttention3 in PSNR across all four tested video DiTs
- Kernel is training-free and composable with sparse attention, distillation, and multi-GPU execution
- Tags
- attention-kernelinference-optimizationlow-bitnunchux-aiquantizationvc-attentionvideo-diffusion-transformersv-smooth
- Vendors
- Nunchux AI
From the original article
4 min read at marktechpost.com
Nunchux AI has released VC-Attention, a training-free low-bit attention kernel built for video Diffusion Transformers (DiTs). It targets 2 problems at once: value quantization error and a slow softmax stage. Why Attention is the Video Bottleneck Video DiTs flatten a clip into 1 sequence of spatiotemporal tokens and run full self-attention at every layer.
By Asif Razzaq · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.