The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends
A survey frames LLM attention as contextual memory across 14 model lineages and 59 releases.
This survey treats efficient attention variants as model-internal contextual memory spanning compression, sparse access, recurrent state, structured dynamics, and heterogeneous composition. It proposes a five-dimensional lens: Memory Representation, Memory Update, Access, Readout, and Integration. The authors reconstruct adoption using 59 release-level records from 14 major model lineages and 11 high-performing open-weight endpoints. They argue sequence architecture is shifting from a single attention operator toward organizing, lifecycle-managing, and selectively routing contextual memory.
- Five-axis lens covers representation, update, access, readout, and integration.
- Evidence drawn from 59 releases across 14 model lineages.
- Depth increasingly coordinates complementary memory mechanisms.
- Sparse write and sparse read decide what each query uses.
From the original article
1 min read at huggingface.co
Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context length. Research thus spans explicit-memory compression, sparse access, recurrent state construction, structured state dynamics, and heterogeneous mechanism composition.
By Zhentao Tan, Jingyi Shen, Yanbo Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.