Skip to content
NewA calmer, faster ZeroHour

SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

arXiv cs.CRby Xiaonan Luo ()1 min read1 · 1 read
AI summary · grok-4.7Summaries by Vani AI

SecProbe adaptively tests coding agents on synthesized vulnerability repairs; frontier models peak at 28.33% success.

SecProbe combines item response theory with on-demand synthesis of repository-scale vulnerability-repair tasks to estimate coding-agent ability and choose the most informative next task. A case study builds 353 tasks spanning six programming languages and 151 CWE types and evaluates nine frontier models with two agent harnesses. Peak success is 28.33%. Compared with random and one-shot baselines, ability estimates stay comparable while agents solve up to 29.5% fewer tasks.

Key facts
  • Constructs 353 repository-scale tasks across six languages and 151 CWE types.
  • Nine frontier models with two harnesses peak at 28.33% success.
  • Matches baseline ability estimates while solving up to 29.5% fewer tasks.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale.

arXiv cs.CR

By Xiaonan Luo · arxiv.org

Read the full article at arXiv cs.CR

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.