An Empirical Study of Harness Design for Coding Agents
Empirical study of coding agent harnesses across 176 settings shows context management prevents overflow failures and bash-only interfaces cut costs for capable models.
The study uses a lightweight coding harness with a fixed execution loop while varying planning, action space, and context management, evaluating 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1. Context management becomes more valuable as context-window budgets tighten, mostly by preventing context-overflow failures, with rule-based elision before LLM summarization the strongest strategy. Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger ones, and bash-capable models work well with a bash-only interface at substantially lower cost, informing model- and budget-aware harness design.
- 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1
- Context management mainly prevents context-overflow failures as window budgets tighten
- Rule-based elision before LLM summarization gives the best efficiency among strategies
- Bash-capable models perform well with bash-only tools at substantially lower cost
- Planning matters more for weaker models' accuracy; stronger models use it to save cost
From the original article
1 min read at arxiv.orgCoding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear.
By Run-Ze Fan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.