VulnBench 2.0: SAST and AI agents miss different web flaws
Original title: How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0
Snyk Code and Claude Code with Opus 5 together located 101 of 108 curated flaws, the ceiling of seven comparable setups.
VulnBench 2.0 scores deterministic SAST against productized AI security agents on 108 hand-curated, attacker-reachable flaws in 20 open-source web apps written in Go, JavaScript/TypeScript, Python and Java. A hit must place the attacker-controlled source and the dangerous sink within two lines. Snyk Code reached 82.4% macro recall (89 of 108) and Claude Code with Opus 5 at xhigh effort 79.3% (85 of 108); they split 28 disagreements 16 to 12. The authors say AI runs also reported at least seven real flaws absent from the reference set and from Snyk Code. Harness, fixtures, reference sets and run data are public.
Unconfirmed: The seven extra findings are the authors' claim that they are real vulnerabilities outside the reference set.
- Sixty-two of the 108 vulnerabilities span more than one file.
- Nine productized configurations were run under one blind protocol.
- Without Claude Code's review policy, the same model matched 16 of 20 restricted-class flaws instead of 9.
- Codex Security with GPT-5.6 Sol found 80 flaws at file level but 34 at line level.
- DeepSec's model-free file selection retained a file from only 56 of 108 flows.
- Vendors
- Snyk
From the original article
1 min read at arxiv.orgWe ask what AI coding agents and AI security harnesses find that deterministic static application security testing (SAST) does not, and what they miss that SAST catches. VulnBench 2. 0 is a benchmark of 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications in Go, JavaScript/TypeScript, Python and Java; 62 flow across more than one file.
By Liran Tal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.