MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
MC-Sparse narrows the dense-sparse attention gap, speeding diffusion denoising up to 2.32× with little quality loss.
Sparse attention often reduces quality in long-sequence diffusion transformers used for video and high-resolution 3D generation. The authors attribute the loss to token grouping, inaccurate interaction selection, and discarded attention contributions. Their training-free Meta-Cached Sparse Attention (MC-Sparse) selects individual KV tokens, groups similar queries into GPU-friendly tiles, and reuses cached metadata and residuals across denoising steps. Versus dense attention, it reports a 1.80× denoising speedup on Minimax-H3-Base and 2.32× on 3D asset generation, with higher fidelity than prior sparse baselines and negligible quality loss.
- Training-free sparse attention for diffusion transformers
- Caches query groups, exact KV indices, and dense-sparse residuals
- 1.80× denoising speedup on Minimax-H3-Base
- 2.32× speedup on 3D asset generation with negligible quality loss
Full article174 words · extracted from huggingface.co · click to collapse
Sparse attention is a primary approach to reducing the latency of diffusion transformers in long-sequence generation tasks, such as video and high-resolution 3D asset generation. However, existing methods can degrade generation quality and fidelity at high sparsity levels. Through controlled oracle comparisons, we trace this degradation to three sources: constraints imposed by token grouping, inaccurate interaction selection, and the attention contributions lost when tokens are discarded. Guided by this analysis, we propose Meta-Cached Sparse Attention (MC-Sparse), a training-free framework that selects individual key-value (KV) tokens while organizing similar queries into tile-aligned groups for efficient GPU execution. MC-Sparse caches metadata comprising query groups, KV indices selected using exact attention probabilities, and residuals between dense and sparse attention outputs, and reuses them across subsequent denoising steps. Across video and 3D generation models, MC-Sparse achieves higher fidelity to dense-attention outputs and larger denoising speedups than existing sparse-attention baselines, without visible quality degradation. Relative to dense attention, it delivers a 1.80times denoising speedup on Minimax-H3-Base and a 2.32times speedup on 3D asset generation, both with negligible quality loss.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.06801