VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities
Introduces VEX-Bench, 75 expert-labeled real-world cases testing whether LLM agents can assess supply chain vulnerability exploitability; frontier models reach about 80% F1.
VEX-Bench is the first benchmark evaluating LLM agents on assessing whether upstream dependency vulnerabilities are exploitable in downstream projects, with 75 real-world expert-labeled cases across Python, Java, and Go mined from GitHub. Nine models across three agent harnesses were evaluated; GPT-5.5 and Claude Opus 4.6 reach approximately 80% F1 on binary vulnerability-status classification, but only GPT-5.5 surpasses 70% macro-F1 on fine-grained justification classification. The gap highlights the difficulty of moving beyond binary exploitability calls to explaining exploitability reasons, unlike prior benchmarks targeting zero-day settings.