An Empirical Study of Harness Design for Coding Agents
Empirical study of 176 matched coding-agent harness configurations shows context management cuts overflow failures while planning and action-space choices trade accuracy against cost.
The paper studies coding-agent harnesses with a fixed execution loop while varying planning, action space, and context management across four models on SWE-Bench Verified and Terminal-Bench 2.1, totaling 176 matched settings. Findings show context management chiefly prevents context-overflow failures, rule-based elision before LLM summarization is most efficient, planning shifts from accuracy scaffold to cost saver as models strengthen, and bash-only interfaces suit bash-capable models at lower cost. Trajectory-level analysis explains how each component alters agent behavior and execution trajectories.
- Evaluates 176 matched harness settings across four models on SWE-Bench Verified and Terminal-Bench 2.1.
- Context management mainly prevents context-overflow failures and matters most under tight context-window budgets.
- Rule-based elision before LLM summarization gives the best efficiency; recoverable elided content rarely helps.
- Planning helps weak-model accuracy but mostly saves cost for strong models.
- Bash-capable models run effectively with bash-only tool interfaces at substantially lower cost.
From the original article
1 min read at huggingface.co
Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear.
By Run-Ze Fan, Zihao Zhang, Simin Ma · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.