MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
Training-free MC-Sparse speeds diffusion-transformer denoising up to 2.32× with negligible quality loss.
Meta-Cached Sparse Attention (MC-Sparse) is a training-free method that narrows the quality gap of sparse attention in diffusion transformers for video and high-resolution 3D generation. It selects individual key-value tokens, groups similar queries into tile-aligned blocks, and caches query groups, exact-probability KV indices, and dense-versus-sparse residuals for reuse across denoising steps. Relative to dense attention it reports a 1.80× denoising speedup on Minimax-H3-Base and 2.32× on 3D asset generation, with negligible quality loss and higher fidelity than prior sparse-attention baselines.
46