Study measures how harness and context change LLM vulnerability discovery
Original title: How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery
No single setup covered all 265 confirmed root causes, and CodeQL context helped only the smallest of four models.
Researchers compared 24 model-runner configurations using an open-source harness on a production codebase whose vulnerabilities were non-public at scan time. Across 265 confirmed security root causes, the largest two-run configuration covered 60.4 percent, while 14.3 percent were found by exactly one model. In a four-model experiment, an engineered prompt improved yield for every model, but adding a CodeQL-derived call graph beyond that prompt helped only the smallest model. Frontier models had low false-report rates but did not consistently lead on yield.
- The pipeline includes static screening, self-revalidation, and isolated proof-of-concept confirmation.
- Moving to an engineered prompt plus CodeQL raised yield 20-86 percent for smaller and mid-tier models.
- The same combined treatment raised yield only 2 percent for the frontier model.
- A held-out CyberGym subset showed the same direction of target-recall improvement.
- Strong lower-cost models can match performance at about an order of magnitude lower cost.
From the original article
1 min read at arxiv.orgHow much does an agent harness affect vulnerability discovery, and does its value depend on the model? We study 24 model-runner configurations using an open-source harness on a production codebase whose vulnerabilities were non-public at scan time.
By Ganni Galea Curmi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.