ExecCritic: Learn to Test, Test to Improve for Coding Agents
ExecCritic separates test generation from patching for coding agents, lifting SWE-bench Verified resolution to 72.6%.
ExecCritic pairs a test-verify-revise scaffold with role-specific reinforcement learning: a Test agent writes repository-native tests and a Repair agent fixes code from execution feedback, both using Qwen-3.5-35B-A3B backbones. Post-trained Qwen agents compose to 72.6% on SWE-bench Verified, an 11.4-point gain over the 61.2% no-test baseline, without stronger-model or oracle feedback at evaluation time. The work shows test quality is the key variable: base-agent tests lowered resolution to 57.3% while GPT-5.6-sol tests raised it to 65.3%.
- Test agent learns behaviorally valid tests; Repair agent learns direct resolution and feedback-guided revision
- Role-specific post-training raised the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%
- Composed post-trained Qwen agents reach 72.6% on SWE-bench Verified
- Code released at github.com/MSR-Orchard/execcritic
- Organizations
- Microsoft Research
- AI models
- Qwen-3.5-35B-A3BGPT-5.6-sol
From the original article
1 min read at arxiv.orgExecution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence.
By Leitian Tao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.