Stratified Sparse Attention narrows the sparse-to-dense gap
Original title: Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
On Ministral-3-8B at 50x sparsity, SSA lifts RULER-HARD from 63.8% to 69.3% at 128K.
Researchers introduce Stratified Sparse Attention, which retrieves high-scoring KV tokens and samples the residual with proxy scores and inclusion-probability correction. It treats top-k, top-p, and hybrid top-k plus sampling as special cases. Across RULER, LongBench, LOFT, and AIME, and five models from 3B to 27B, the authors say SSA beats baselines at matched sparsity. Optimized CUDA kernels reach up to 2.96x decode speedup over FlashInfer at 1M context. Code is open-sourced.
Unconfirmed: Accuracy and speedup figures are the authors' measurements.
- On RULER-HARD with Ministral-3-8B, 32K accuracy rises from 81.8% to 84.2%.
- At 128K under 50x sparsity, accuracy rises from 63.8% to 69.3%.
- Those runs recover 94% and 85% of dense-attention performance.
- Decode speedup is up to 2.96x versus FlashInfer at 1M context.
- Models tested range from 3B to 27B parameters.
- Products
- FlashInfer
- AI models
- Ministral-3-8B
From the original article
1 min read at huggingface.co
Large Language Model inference over long contexts is increasingly bottlenecked by the cost of attending to a growing Key-Value (KV) cache.
By Sahil Joshi, Prithvi Dixit, Aryan Katiyar · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.