Small coding agents gain more from code graphs than larger models
First merged summary.
On SWE-bench Verified, structured tools add 17–21 points for 1.7–4B agents but only 4–5 at 32B.
An arXiv study ablates grep, embedding search, and code-graph tools for small open-weight coding agents on SWE-bench Verified, stratified by hop depth. Structured tools doubled grep-only accuracy for Qwen3-8B and Gemma 4 E4B, adding 17–21 points at 1.7–4B parameters versus 4–5 points at 32B. Ten models and 79,000 episodes were evaluated, including six Qwen3 sizes and three Gemma 4 sizes. Agents mostly used name lookup and a single expansion; each extra hop cut correct-localization odds at every size. The authors release the agent, probes, logs, and traces.
- Structured tools added 17–21 points at 1.7–4B and 4–5 points at 32B.
- Tools doubled grep-only accuracy for Qwen3-8B and Gemma 4 E4B.
- Ten models and 79,000 episodes covered six Qwen3 and three Gemma 4 sizes.
- Each extra hop multiplied correct-localization odds by 0.50–0.67.
- Agents chained graph expansions in only 12–14% of episodes.
- A fixed repair model resolved 6–13 of 160 tasks; the oracle resolved 26.
What each outlet adds
beyond the shared facts- arXiv cs.AI / cs.LG / cs.CLHop-depth ablation, localization odds, and release of agent, probes, logs, and traces.
Coverage
oldest first · one row per article · UTC- · 6d ago
Hugging Face daily papersFirst report
Independent review lifts coding-agent accuracy at modest extra cost
A critic review gains 5.9–9.5 points on Terminal-Bench and 3.9–6.6 on SWE-bench Verified at about baseline cost.
- · 5d ago
arXiv cs.AI / cs.LG / cs.CL
Small coding agents gain little from code graphs past one hop
On SWE-bench Verified, structured tools added 17–21 points at 1.7–4B but only 4–5 points at 32B.