Terminal-Bench 2.1 repairs improve coding agent benchmark accuracy
Original title: Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
Benchmark maintenance debt leads to issues in 30% of tasks with accuracy gains over 10 percentage points after repairs.
As models improve, benchmarks accumulate defects known as maintenance debt. A continuous-validation workflow applied to Terminal-Bench 2.0 identifies issues in roughly 30% of tasks and releases repairs as Terminal-Bench 2.1. Accuracy increases by over 10 percentage points and leaderboard positions shift, while resource settings such as time limits affect performance gaps.
- Issues identified in roughly 30% of benchmark tasks
- Accuracy increased by over 10 percentage points after repairs
- Leaderboard positions move by up to three ranks
- Spearman rho remains 0.958
- Time limits can close 20pp or 30pp gaps on GPT models
From the original article
1 min read at arxiv.orgCoding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers reject, while changes to evaluation infrastructure can invalidate assumptions embedded in tasks and tests.
By E. Kelly Buchanan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.