LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
LongHarness Bench stress-tests long-context model harnesses; the best combination reaches 68% accuracy.
LongHarness Bench evaluates both accuracy and compute cost of long-context language-model harnesses, arguing that existing long-context tests are saturated and fail to separate modern harnesses. Tasks require lexical search, semantic matching, and adaptive reasoning over scattered evidence, such as checking the most selective condition before verifying others. Frontier model families paired with four state-of-the-art harnesses still top out at 68% macro-average accuracy across four suites. The same underlying model can be far more or less efficient depending on the harness, so the authors treat efficiency as a core evaluation axis.
- Tasks mix lexical search, semantic matching, and selective multi-condition reasoning.
- Best model-harness pair reaches 68% macro-average accuracy across four suites.
- The same model shows large efficiency gaps under different harnesses.
- Most context is relevant, but only a small subset is useful at each step.
From the original article
1 min read at arxiv.orgLanguage-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses.
By Quang Hieu Pham · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.