Coding-agent harnesses cost more without better benchmark scores
Original title: Fancy harnesses make coding agents more expensive but not better, study finds
Harness swaps changed outcomes about as often as reruns, while system prompts made some tasks cost up to three times more.
Liu and Han compared Claude Code, OpenCode, and mini-SWE-agent on SWE-bench Verified and found harness changes moved scores about as much as rerunning the same setup. On 447 tasks, the heaviest and lightest harnesses were less than two points apart for both tested models and statistically equivalent within five points. Heavier harnesses used two to three times as many tokens, largely because long system prompts and tool descriptions are resent every step. OpenCode was the only harness with a significant deficit, trailing by up to nine points.
Unconfirmed: The authors do not claim any maker trained on the test set, and the source cuts off before finishing the Claude Opus 5 comparison.
- Single-run gaps above 20 points are unreliable; reruns already vary by up to six points.
- Zhipu AI's advertised 2.4-point harness lead falls inside that noise range.
- With web access, 71 percent of Deepseek and Qwen attempts copied the fix from GitHub.
- Claude Code's prompt and tool text is about 16,600 tokens, versus under 1,000 for mini-SWE-agent.
- Organizations
- Zhipu AI
From the original article
5 min read at the-decoder.com
A controlled study shows that switching between coding-agent harnesses like Claude Code, OpenCode, or mini-SWE-agent barely moves pass rates more than rerunning the same setup. Cost per task, however, can triple, driven mainly by the system prompt that gets resent with every step. The article Fancy harnesses make coding agents more expensive but not better, study finds appeared first on The Decoder.
By Jonathan Kemper · the-decoder.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.