On-Demand Attention: Language Models Know When to Recall
On-Demand Attention uses a lightweight recall head to selectively invoke global attention, cutting long-context decoding costs in vLLM with minimal quality loss.
Researchers introduce On-Demand Attention (ODA), a local-first decoding method in which a lightweight recall head predicts when global attention benefits the next token. Only the recall head is trained, leaving pretrained weights and the complete KV cache unchanged. GPU-side conditional execution implemented in vLLM converts reduced global reads into practical decoding speedups at long context lengths. Experiments across Qwen and Gemma models, including hybrid-attention backbones, show selective recall recovers most of the performance lost under local attention.
- Decoding states predict whether global attention helps the next prediction
- Trains only the recall head; pretrained weights stay frozen
- Conditional execution implemented in vLLM for practical decoding speedups
- Validated on Qwen and Gemma models including hybrid-attention backbones
- Products
- vLLM
From the original article
1 min read at arxiv.orgReasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. We show that a pretrained model's decoding states already contain information predictive of this benefit, before the global read.
By Haibo Feng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.