SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS trains attention sparsification end-to-end with the language modeling loss, beating sparse attention baselines especially under tight context budgets.
Simple Attention Sparsification (SAS) injects the selector's continuous scores into attention logits in log form inside the softmax, letting gradients from the language modeling loss directly update the ranking of context units. The method uses normalized softmax gates calibrated against the current block and a memory-efficient Triton kernel integrated into FlashAttention-style computation. Across reasoning, long-context, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across budgets, with the largest gains under tight attention budgets.
- End-to-end optimization aligns context ranking with prediction impact, not dense attention distillation
- Continuous gated scores inside the softmax enable gradient flow to the selector
- Memory-efficient Triton kernel supports long-sequence training
- Largest gains over baselines appear under tight attention budgets
From the original article
1 min read at huggingface.co
Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods usually use a lightweight selector to score context units, followed by hard Top-K selection that blocks gradients from the language modeling loss.
By Zhiwei Li, Lei Zhu, Hao Gu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.