Independent review lifts coding-agent accuracy at modest extra cost
Original title: Rethinking Verification in Agent Harnesses
A critic review gains 5.9–9.5 points on Terminal-Bench and 3.9–6.6 on SWE-bench Verified at about baseline cost.
A paper studies verification in agent harnesses as test-time scaling across initial instructions, mid-turn reminders, completion gates, and independent review. It compares 17 designs on seven frontier coding models using Terminal-Bench, SWE-bench Verified, and a new VisualBench. Independent review gives the better accuracy-cost tradeoff; extra review rounds raise average Terminal-Bench accuracy by up to 3.5 points.
- Completion gates can raise accuracy but may substantially increase cost.
- Instructions and reminders give modest, model-dependent gains without much extra token use.
- VisualBench independent review wins by 13.2–23.8 net preference points.
- Critic review costs 1.13–1.25 times baseline on Terminal-Bench and 1.00–1.04 on SWE.
From the original article
1 min read at huggingface.co
Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort.
By Zhengxin Zhang, Erkang Zhu, Yin Lin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.