SPLASH localizes insecure statements in LLM-generated code without training
Original title: SPLASH: Contrastive Audit Pressure for Detecting and Localizing Security Hallucinations in Generated Code
Across 63,684 programs from 12 models, it selected secure code in 74% of cases and ranked the vulnerable statement first 58% of the time.
SPLASH targets security hallucinations: code that passes functional tests while remaining insecure. It contrasts teacher-forced log-probabilities of a program under a security review and a style review, a measure the authors call audit pressure. The method needs no training, CWE-specific rules or internal access to the generating model, and the authors say it beat a frontier LLM judge.
Unconfirmed: The selection and localization rates are results reported by the paper's authors.
- The evaluation covered 899 tasks and 12 generating models.
- It placed the vulnerable statement first in 58% of insecure programs.
- Audit pressure is aggregated to both statements and whole programs.
- No CWE-specific rules or generator-model internals are required.
From the original article
1 min read at arxiv.orgCode generated by large language models often passes every functional test while remaining insecure. These security hallucinations are common, written with high confidence, and repeated across samples, so neither the generator's own uncertainty nor static analyzers reliably catch them. We introduce SPLASH, which is guided by a new measure called audit pressure.
By Mirza Masfiqur Rahman · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.