SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities
SecProbe adaptively tests coding agents on synthesized vulnerability repairs; frontier models peak at 28.33% success.
SecProbe combines item response theory with on-demand synthesis of repository-scale vulnerability-repair tasks to estimate coding-agent ability and choose the most informative next task. A case study builds 353 tasks spanning six programming languages and 151 CWE types and evaluates nine frontier models with two agent harnesses. Peak success is 28.33%. Compared with random and one-shot baselines, ability estimates stay comparable while agents solve up to 29.5% fewer tasks.
- Constructs 353 repository-scale tasks across six languages and 151 CWE types.
- Nine frontier models with two harnesses peak at 28.33% success.
- Matches baseline ability estimates while solving up to 29.5% fewer tasks.
From the original article
1 min read at arxiv.orgAssessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale.
By Xiaonan Luo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.