Small coding agents gain little from code graphs past one hop
Original title: How Do Small Coding Agents Use Code Graphs and Embeddings?
On SWE-bench Verified, structured tools added 17–21 points at 1.7–4B but only 4–5 points at 32B.
The study ablates grep, embedding search, and code-graph tools for small open-weight coding agents on SWE-bench Verified, stratified by hop depth. Structured tools doubled grep-only accuracy for Qwen3-8B and Gemma 4 E4B, but gains shrank as models grew. Agents mostly used name lookup and a single expansion; extra hops still cut localization odds at every size. The authors release the agent, probes, logs, and traces.
- Ten models and 79,000 episodes were evaluated, including six Qwen3 sizes and three Gemma 4 sizes.
- Each extra hop multiplied correct-localization odds by 0.50–0.67 at every size.
- Agents chained graph expansions in only 12–14% of episodes.
- A fixed repair model resolved 6–13 of 160 tasks; the oracle resolved 26.
- Direct call edges probed at AUC 0.74, below a 0.99 lexical baseline.
- Products
- SWE-bench Verified
From the original article
1 min read at arxiv.orgCoding agents are given embedding search and code-graph tools on the premise that a repository's structure is something the model lacks and the tool supplies. We test that premise on SWE-bench Verified with small open-weight models, stratifying tasks by a deterministic hop depth (the shortest code-graph path from code the issue mentions to code the gold patch changes) that a companion paper defines and validates.
By Saurav Bellam · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.