VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
VC-Attention is a training-free low-bit attention method for diffusion transformers, achieving 1.46-1.59x kernel speedups on datacenter GPUs with higher fidelity.
VC-Attention is a training-free low-bit attention framework for diffusion transformers that pairs V-Smooth value smoothing via lightweight online clustering with ExpCast-FP8, which maps log-domain scores directly to E4M3 FP8 probability codes and eliminates the FP32 softmax exponential. It is implemented for B200, B300, H200, RTX PRO 6000, and RTX 5090 GPUs. Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, it improves fidelity over low-bit baselines and speeds attention 1.46-1.59x over BF16 FlashAttention-4 on datacenter Blackwell and Hopper GPUs and 2.3-3.6x on workstation cards, with 1.13-1.70x faster end-to-end clip generation.
- Training-free: works with existing diffusion transformer checkpoints without retraining.
- V-Smooth reorders value tokens via online clustering to reduce quantization error.
- ExpCast-FP8 removes the FP32 softmax exponential via fused FP8 probability casting.
- 1.46-1.59x attention speedup on datacenter GPUs; 2.3-3.6x on workstation GPUs.
- Tags
- diffusion-transformersfp8gpu-kernelsinference-optimizationlow-bit-attentionquantizationvideo-generation
- Vendors
- NVIDIA
- Products
- VC-AttentionFlashAttention-4
From the original article
1 min read at huggingface.co
Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical entries confined to a narrow range of representable values.
By Xingyang Li, Dongyun Zou, Shining Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.