Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer
A coding-agent benchmark finds compact documentation does not help agents fix real repository issues.
The authors introduce a roundtrip benchmark that scores natural-language code descriptions by whether code regenerated from them passes the original tests, and find that completeness rather than length drives fidelity. Using that score as an optimization signal, they discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. Across two model families and ten repositories, neither static compact documentation nor retrieved context beats the issue text alone when the source is present, even though a positive control shows the evaluation can detect a real improvement.
- Roundtrip tests check whether code regenerated from a description passes original tests.
- Completeness, not length, drives a description's fidelity.
- An optimized writing prompt reaches full fidelity on unseen files.
- Compact docs did not beat the issue alone when source code was present.
From the original article
1 min read at arxiv.orgWe investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity.
By Md Shohel Arman · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.