Attention competition predicts context rot in retrieval
Original title: Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
Filler tokens maintain accuracy up to 32K tokens in retrieval but rival statements multiply loss by 9-110 across five open models.
Language models fail on long inputs due to attention competition in retrieval. Neutral fillers keep accuracy at 0.97-1.00 up to 32K tokens on Qwen2.5-1.5B and other models. Statements giving rival key values multiply loss by 9 to 110. Attention mass on evidence explains up to 58% of loss variance. The masked-filler protocol fails in retrieval control.
- Neutral fillers preserve accuracy at 0.97-1.00 up to 32K tokens
- Rival statements drop accuracy to 0.16-0.33
- Attention mass left on evidence explains up to 58% variance
- Masking costs 45 points in retrieval but not in MMLU
- Organizations
- huggingface
- AI models
- qwen2.5-1.5b
From the original article
1 min read at huggingface.co
Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not.
By Yinuo Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.