The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence
LYRA realigns long-context attention so distant task evidence is not drowned out by proximal background.
The paper argues long-context LLMs miss distant evidence less because of distance than because abundant task-irrelevant nearby text competes for attention, a failure it calls the Proximity Trap. LYRA (Long-context heavy-tailed Relevance Alignment) uses t-distributed directional matching to shift retrieval mass toward relevant spans while preserving relative position. Experiments on LongBench-v2, RULER, and LongBench report consistent gains across lengths and tasks, and the authors introduce ProxBench to test distant-evidence use under increasing proximal interference.
- Proximity Trap: nearby irrelevant text crowds out distant evidence.
- LYRA uses t-distributed directional matching to reweight retrieval.
- Gains reported on LongBench-v2, RULER, and LongBench.
- ProxBench measures distant-evidence use under proximal interference.
From the original article
1 min read at arxiv.orgLong-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background.
By Xiaoyu Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.