On the Reliability of LLM-Based Vulnerability Patching Benchmarks
Study finds LLM vulnerability-patching benchmarks inflate success while developer-aligned fixes stay rare across 112 bugs.
The authors argue that LLM vulnerability-patching benchmarks can distort results through agent prompts and tools, framework failures, and weak tests. They curate 112 historical bugs from 84 open-source C/C++, Go, and Rust projects, each with proof-of-concept, regression, and developer-intent tests. Models can achieve high PoC pass rates under ideal conditions, but execution choices materially change measured success. Developer-test pass rates remain low and improve only marginally with newer models, suggesting symptom suppression rather than upstream-quality fixes.
- Curates 112 bugs from 84 C, C++, Go, and Rust projects
- Prompts, tools, and instructions can inflate PoC success
- Infrastructure bugs and timeouts can suppress or inflate scores
- Developer-intent tests stay low even as models get newer
From the original article
1 min read at arxiv.orgLarge language models (LLMs) have shown strong potential for automated vulnerability patching, but current benchmarks can substantially distort reported performance.
By Dang K Le · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.