How well do agents use test/verification techniques?
Dan Luu's eval finds coding-agent testing instructions (TDD, formal methods, PBT, skills) mostly fail to beat defaults on Zstd implementation correctness.
The author ran 26 prompt conditions plus 4 skills on a Zstd-in-Rust implementation eval using codex with GPT-5.6, testing TDD, fuzzing, property-based testing, formal methods (Lean 4, TLA+, Verus, Kani, SMT solvers) and community skills. Nothing dramatically outperformed the default no-instruction condition, which did above average; at xhigh effort, fuzzing and PBT conditions did slightly better than formal methods. Pre-registered predictions included TDD underperforming and popular test skills (ECC, Hegel, Trail of Bits) not outperforming. Results are averages of 80 runs per condition plotted against cost.
- 26 testing techniques and 4 skills evaluated on Zstd-in-Rust task
- Default no-instruction condition performed above average
- Fuzzing/PBT edged out formal methods at xhigh effort
- Community test skills (ECC, Hegel, Trail of Bits) underperformed
- Predictions pre-registered with stated confidence levels
- Organizations
- Trail of Bits
- AI models
- GPT-5.6
From the original article
15 min read at danluu.comWe previously noted that, while it's easier than ever to hit a particular quality bar by having coding agents use effective test techniques, software quality seems to be getting worse , indicating that whatever defaults developers are using may not work very well.
By vinhnx · danluu.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.