Coding-agent harness value shifts when the model changes
Original title: In 27,280 coding agent runs, 64% of inference spend went to runs that failed
In 27,280 runs using 83 billion tokens, 64% of inference spend went to runs that failed.
Runta and Baseten expanded FrontierHarness Eval to 22 harness configurations, five models, 62 tasks and four attempts. The benchmark found that the cheapest or best-value harness changed with the model, while the model itself explained most of the cost difference. Over all models, OpenCode v2 had the lowest average cost per task and ZCode the highest pass rate.
Unconfirmed: Except for Codex CLI with GLM-5.3, the best-value winners were close and less stable under resampling.
- Best-value pairings included Codex CLI with GLM-5.3 and Alma with Kimi-K3.
- Codex CLI ranked first by cost with GLM-5.3 but 16th with Kimi-K3.
- OpenCode v2 averaged $0.561 per task; ZCode’s pass rate was 61.9%.
- Codex CLI remained first with GLM-5.3 in 99.6% of 4,000 task resamples.
- Audits scored 216 reward-hacking trials as zero, including 163 online lookups.
- Products
- FrontierHarnessHarbor
From the original article
11 min read at runta.com
We partnered with Baseten to build the largest-scale harness leaderboard using 83B tokens across 27,280 agent runs. We wanted to answer two questions: when you switch your model, should you consider switching the harness too? What is the best harness for each model?
By Edward40 · runta.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.