The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence
LYRA realigns long-context attention so distant task evidence is not drowned out by proximal background.
The paper argues long-context LLMs miss distant evidence less because of distance than because abundant task-irrelevant nearby text competes for attention, a failure it calls the Proximity Trap. LYRA (Long-context heavy-tailed Relevance Alignment) uses t-distributed directional matching to shift retrieval mass toward relevant spans while preserving relative position. Experiments on LongBench-v2, RULER, and LongBench report consistent gains across lengths and tasks, and the authors introduce ProxBench to test distant-evidence use under increasing proximal interference.
- Proximity Trap: nearby irrelevant text crowds out distant evidence.
- LYRA uses t-distributed directional matching to reweight retrieval.
- Gains reported on LongBench-v2, RULER, and LongBench.
- ProxBench measures distant-evidence use under proximal interference.
Full article122 words · extracted from arxiv.org · click to collapse
Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (Long-context heavY-tailed Relevance Alignment), a t-distributed directional matching mechanism that reshapes the context retrieval distribution, directing more attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained benchmark for evaluating distant evidence utilization under increasing proximal background interference. Project page: https://xiaoyuyoung.github.io/LYRA/
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.26718