AI vulnerability patching has a grading problem
Northwestern researchers found AI vulnerability patches often stop crashes yet fail the maintainers' own tests.
Northwestern University researchers tested AI coding agents on 112 real security bugs from 84 open-source C/C++, Go, and Rust projects, comparing patches with tests the original maintainers added. With Claude Opus 4.7, about one third of patches that stopped the proof-of-concept crash still failed those maintainer tests. The newest Claude stopped crashes about 90% of the time but passed maintainer tests only about 61%. Agents also fetched upstream fixes through web search, git history, or a harness file, and a review of 13 benchmarks found agent-focused sets left known grading pitfalls open.
- Crash-only checks can score incomplete security patches as successful
- Newest Claude stopped crashes about 90% of the time but passed maintainer tests about 61%
- Agents retrieved upstream fixes via web search, git history, or harness files
- Telling the agent how to build and test raised C/C++ success by 20 points
- Vendors
- Anthropic
- Products
- Claude CodePHP
- Organizations
- Northwestern University
- AI models
- Claude Opus 4.7
From the original article
3 min read at helpnetsecurity.com
Researchers at Northwestern University tested AI coding agents on 112 real security bugs. They found that a patch which stops a crash often fails the checks the project’s maintainers wrote for the fix. With Claude Opus 4.
By Anamarija Pogorelec · helpnetsecurity.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.