Newer and Bigger, but Safer? A Longitudinal Study of the Functionality-Security Gap in LLM-Generated Code
Across 32 LLMs, newer models are safer in absolute terms, but no family closes the code security gap.
A longitudinal study evaluates the functionality-security gap across 32 LLMs from seven families, five open-weight, with three successive releases in flagship and compact variants. Using CWEval's 119 tasks in five languages and 31 CWEs, newer models are safer in absolute terms, but no family closes the gap between passing functional tests and failing security tests. Unlike earlier work, every considered open-weight family narrows the gap, while Gemini 3.1 Pro remains as wide as previously reported for Llama. Compact models are usually less secure than flagships, with exceptions such as Gemini 3.7 Flash, and CWE-117 and CWE-113 persist while some proprietary models regress on memory and integer weaknesses.
- Thirty-two models from seven families; no family closes the security gap.
- Every studied open-weight family narrows the gap significantly.
- Gemini 3.1 Pro keeps a gap as wide as previously reported for Llama.
- CWE-117 and CWE-113 remain persistent, with risk varying by language.
- Products
- CWEval
- AI models
- Gemini 3.1 ProGemini 3.7 FlashLlama
From the original article
1 min read at arxiv.orgLarge Language Models (LLMs) are widely used to generate code. Although their functional plausibility keeps improving, the generated code often contains security vulnerabilities. The functionality-security gap captures code that passes functional tests but fails security tests.
By Thiago Santos de Moura · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.