Paper treats benchmarks as difficulty measures and proposes BenchFill
Original title: Your Benchmark Is an Empirical Measure Over Difficulty
Authors diagnose mismatch and coverage gaps on 25 benchmarks across seven domains, then rewrite items to fill gaps.
A paper argues that a benchmark induces an empirical measure over difficulty, not just a set of questions. It formalizes two failure modes: measure mismatch, when the difficulty distribution differs from the intended one, and coverage gaps, when uncovered difficulty regions hide localized training progress. Two diagnostic metrics with theoretical guarantees are proposed. On 25 popular benchmarks across seven domains, the authors say many overweight some difficulty regions and others leave large stretches uncovered. BenchFill, a difficulty-targeted item-rewriting algorithm, is introduced to reduce local coverage gaps.
Unconfirmed: Difficulty estimates and the claim that BenchFill reduces gaps are the authors' results.
- Diagnostics target measure mismatch and coverage gaps along a difficulty axis.
- Analysis uses estimated item difficulties for 25 benchmarks in seven domains.
- Authors say some leaderboards should be read with caution because of overweighting.
- BenchFill rewrites items to shrink local coverage gaps.
From the original article
1 min read at huggingface.co
A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation.
By Yifan Sun, Naicheng Yu, Jiawen Gong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.