VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities
Introduces VEX-Bench, 75 expert-labeled real-world cases testing whether LLM agents can assess supply chain vulnerability exploitability; frontier models reach about 80% F1.
VEX-Bench is the first benchmark evaluating LLM agents on assessing whether upstream dependency vulnerabilities are exploitable in downstream projects, with 75 real-world expert-labeled cases across Python, Java, and Go mined from GitHub. Nine models across three agent harnesses were evaluated; GPT-5.5 and Claude Opus 4.6 reach approximately 80% F1 on binary vulnerability-status classification, but only GPT-5.5 surpasses 70% macro-F1 on fine-grained justification classification. The gap highlights the difficulty of moving beyond binary exploitability calls to explaining exploitability reasons, unlike prior benchmarks targeting zero-day settings.
- First benchmark for LLM assessment of software supply chain vulnerability exploitability
- 75 real-world GitHub cases across Python, Java, and Go labeled by security experts
- GPT-5.5 and Claude Opus 4.6 reach about 80% F1 on binary classification
- Only GPT-5.5 surpasses 70% macro-F1 on fine-grained justification classification
- Targets known-vulnerability downstream impact reasoning rather than zero-day detection
- Products
- VEX-BenchDependabot
- AI models
- GPT-5.5Claude Opus 4.6
From the original article
1 min read at arxiv.orgThe software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing defenses such as GitHub Dependabot often raise many false alerts because their coarse-grained matching cannot determine whether a vulnerable dependency is actually exploitable.
By Jiahao Shi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.