Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
VLoc Bench tests 27 language models at locating vulnerable files in 290 repositories; best system reaches 0.229 File F1 and 38.4% of tasks unsolved.
The Vulnerability Localization Benchmark (VLoc Bench) contains 500 real-world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories, pairing pre-fix and post-fix repository snapshots. Agents receive only a CWE description and read-only terminal access to identify affected files, and must confirm absence on patched snapshots. The strongest of 27 language models and four static-analysis tools achieves just 0.229 File F1; 38.4% of tasks receive no correct localization, and effective localizers still report unsupported locations on patched repositories.
- 500 real vulnerabilities across 290 repositories, six ecosystems, 147 CWE categories
- Agents localize from CWE description plus read-only terminal on pre-fix snapshots
- Best of 27 models and 4 static-analysis tools reaches only 0.229 File F1
- 38.4% of tasks receive no correct localization from any evaluated model
- Strong localizers still report unsupported locations on patched repositories
From the original article
1 min read at arxiv.orgLanguage-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code. We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness.
By Aman Priyanshu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.