KEEP-Bench finds context compressors often drop required evidence
Original title: KEEP-Bench: Evaluating Evidence Preservation in Context Compression
The best open-weight 31B model keeps all required evidence in 66.8% of 4,102 cases, and the best proprietary model in 61.6%.
Researchers introduce KEEP-Bench to test whether context compression preserves evidence that QA metrics can miss. The benchmark has 4,102 cases covering plausible reference resolutions, explicit qualifiers, and minority perspectives. Even with tailored prompting, inputs averaging 3.1K tokens, and context windows over 100K tokens, the strongest models remain unreliable. Their Evidence Preservation Return metric peaks at moderate 4B–8B scales, where recall stays low.
- Three settings cover reference resolution, explicit qualifiers, and minority perspectives.
- Performance varies sharply across evidence structures, so narrow tests can bias conclusions.
- Extra output budget mainly helps already strong models and widens gaps.
- Authors say preservation gains from those models carry a steep compute cost.
From the original article
1 min read at arxiv.orgContext compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss.
By Ziqing Man · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.