Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Researchers show compile rate poorly measures LLM repairs of C/C++ vulnerabilities and propose a change-aware screen.
An arXiv cs.CR study argues that compile rate is an unreliable metric for single-function C/C++ vulnerability repair by LLMs. Five experiments cover 203 Big-Vul functions, three open-source code models from 350 million to 6.7 billion parameters, and three prompting strategies. About 64 percent of compile failures were not attributable to the model, compiler flags moved the rate by 1.8 to 2.7 times, and optimizing compile rate increased non-repairs. Whole-function CodeBLEU also fails, and the authors offer diff_F1 only as a cheap change-aware screen before execution-based evaluation.
- Study tests compile rate on 203 Big-Vul functions and three code LLMs.
- About 64% of compile failures were not caused by the model.
- One compiler flag shifted compile rate by 1.8 to 2.7 times.
- Optimizing compile rate rewarded deletion and placeholder non-repairs.
- diff_F1 is proposed as a cheap screen, not a quality metric.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles.
By Om Nepal · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.