SecPRBench measures agentic security review on real bug-introducing commits
Original title: Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?
Recall spans 32.5% to 78.1% across eight frontier models on 265 tasks from nine systems, led by Grok 4.6 and GPT-6 sol.
SecPRBench evaluates security-focused code-review agents on validated bug-introducing commits rather than later fixes. An automated pipeline harvests and checks the commits, and an LLM judge credits findings that match known flaws. Most misses name the flawed file but argue for a different mechanism. OpenAI and Anthropic security harnesses shift recall in opposite directions.
- Tasks include Firefox, V8 and OpenSSL.
- In 68% of misses the agent cites the flawed file but the wrong mechanism.
- Codex Security lowers recall by 8.3 points, mostly by abstaining.
- Anthropic's Defending Code harness raises recall by 7.7 points at higher cost.
- Products
- FirefoxV8OpenSSLCodex Security
From the original article
1 min read at arxiv.orgLLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes.
By Yihan Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.