SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

SAS trains a gated sparse attention selector end-to-end with the language modeling loss by injecting continuous selector scores directly into attention softmax logits, outperforming trainable sparse attention baselines on reasoning, long-context, and agentic…

Simple Attention Sparsification (SAS) sparsifies attention by injecting a selector's continuous, gated scores into the attention softmax logits in log form, so that gradients from the language modeling loss directly optimize the ranking of context units. This contrasts with prior Top-K sparsifiers, which train their selectors by distilling dense attention distributions. Key design choices include normalized softmax gates calibrated against the current attention block and preservation of continuous selector scores. A memory-efficient Triton kernel integrates SAS into FlashAttention-style computation, enabling long-sequence training. Across reasoning, long-context, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across budgets, with the largest gains appearing under tight attention budgets. The paper is covered by two independent reports — Hugging Face daily papers (2026-09-10) and arXiv cs.AI/cs.LG/cs.CL (2026-09-11) — which agree on all technical details; no contradictions were found between them.

• Paper title: 'SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking'.
• Core mechanism: the selector's continuous scores are injected into attention softmax logits in log form, letting gradients from the language modeling loss directly update the ranking of context units.
• SAS avoids distilling dense attention distributions, unlike prior Top-K sparse attention methods.
• Gates are normalized and calibrated against the current attention block, and continuous selector scores are preserved for gradient flow.
• A memory-efficient Triton kernel integrates SAS into FlashAttention-style computation to support long-sequence training.
• Evaluated on reasoning, long-context, and agentic tasks, where SAS consistently outperforms trainable sparse attention baselines across budgets.
• The largest gains over baselines occur under tight attention/context budgets.
• Sources: Hugging Face daily papers (2026-09-10) and arXiv cs.AI/cs.LG/cs.CL (2026-09-11); both reports describe the same paper with no factual disagreements.