LSA maintains accuracy at ultra-long context via latent sparse attention
Original title: Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
LSA clusters keys in learned latent space for subquadratic attention, speeding up layers by 22.7x at 10M tokens while matching full-attention accuracy.
Agentic workflows require long context windows for complex tasks, but full attention becomes prohibitive above 1M tokens. Latent Sparse Attention hierarchically clusters keys in latent space and scores clusters instead of every key. Two variants enable application to pre-trained models or native training, offering practical efficiency for ultra-long context.
- Hierarchical clustering in learned latent space
- Prefill/decode/training faster at long context
- D-LSA speeds one layer by 22.7x prefill on GLM-5.3-FLASH at 10M tokens
- N-LSA matches full attention accuracy with 1.6x training speedup
From the original article
1 min read at arxiv.orgAgentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention.
By Ales Kubicek · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.