ZHA achieves subquadratic sparse attention for long-context LLMs
Original title: Subquadratic Attention with a Hierarchical Tree-Structured Indexer
ZHA uses hierarchical tree indexers for subquadratic attention scaling beyond 1M tokens with maintained performance.
Quadratic attention limits LLM long-context capabilities. Existing sparse methods fail to match expressiveness or only offer constant speedups. ZHA constructs recursively compressed key summaries in a tree over KV cache. Queries traverse the tree top-down for sparse key retrieval. Total cost including routing and attention scales sublinearly in sequence length. The method applies to pretrained models via short KL distillation and maintains performance on retrieval and reasoning tasks.
- Subquadratic scaling including routing and attention
- Robust to tree depths up to 6 levels
- Applies via KL distillation on few billion tokens
- Outperforms competitors beyond 1M context
- Extends effective windows to hundreds of millions or billions of tokens
From the original article
1 min read at arxiv.orgThe quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models (LLMs). A variety of approaches have attempted to circumvent this, but they either fail to match attention's expressiveness (e. g.
By Leon Lufkin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.