Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Researchers show compile rate poorly measures LLM repairs of C/C++ vulnerabilities and propose a change-aware screen.
An arXiv cs.CR study argues that compile rate is an unreliable metric for single-function C/C++ vulnerability repair by LLMs. Five experiments cover 203 Big-Vul functions, three open-source code models from 350 million to 6.7 billion parameters, and three prompting strategies. About 64 percent of compile failures were not attributable to the model, compiler flags moved the rate by 1.8 to 2.7 times, and optimizing compile rate increased non-repairs. Whole-function CodeBLEU also fails, and the authors offer diff_F1 only as a cheap change-aware screen before execution-based evaluation.
- Study tests compile rate on 203 Big-Vul functions and three code LLMs.
- About 64% of compile failures were not caused by the model.
- One compiler flag shifted compile rate by 1.8 to 2.7 times.
- Optimizing compile rate rewarded deletion and placeholder non-repairs.
- diff_F1 is proposed as a cheap screen, not a quality metric.
Full article277 words · extracted from arxiv.org · click to collapse
Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.26749