Towards Scalable and Cost-Efficient Vulnerability Detection: A Study on Automatic Query Generation
A study finds LLM-synthesized CodeQL queries improve average F1-score by 82% over baseline queries, offering scalable vulnerability detection versus direct LLM scanning.
Researchers conducted an empirical study evaluating whether LLMs can synthesize executable CodeQL queries from National Vulnerability Database vulnerability data. LLM-generated queries significantly enhanced baseline CodeQL suites, yielding an 82% improvement in average F1-score across a diverse set of real-world vulnerabilities. A cost-benefit analysis shows direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, while LLM query synthesis offers a scalable and cost-effective alternative for large-scale vulnerability detection.
- LLM-generated CodeQL queries built from NVD data raised average F1-score by 82% over baseline query suites.
- Direct LLM scanning of full repositories is often prohibitive; synthesizing static-analysis queries is cost-effective.
- Multiple LLM architectures were evaluated against real-world vulnerabilities for coverage and precision.
- Findings suggest a scalable path from unstructured vulnerability reports to formal static-analysis specifications.
- Vendors
- GitHub
From the original article
1 min read at arxiv.orgStatic analysis remains a cornerstone of software security, yet the effectiveness of tools such as CodeQL is often limited by the substantial manual effort required to develop high-coverage query suites. While large language models (LLMs) have emerged as a potential solution for automated code reasoning, their practical utility in generating structured, executable security queries remains underexplored.
By Ivana Clairine Irsan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.