Learning from Failures: A Failure-Driven Prompt Refinement for LLM-Based Vulnerability Analysis
Failure-driven prompt refinement uses recurring LLM mistakes to improve vulnerability analysis on DVJA and Juliet.
Failure-Driven Prompt Refinement analyzes recurring LLM errors on the Damn Vulnerable Java Application, including false positives, false negatives, unsupported reasoning, and CWE misclassification, then converts those patterns into prompt changes. The refined prompt is evaluated on the Juliet Test Suite and checked across models. The authors report more reliable vulnerability analysis and reusable prompt-design principles rather than a new detector or exploit.
- FDPR turns recurring model failures into targeted prompt edits.
- Modes include false positives, false negatives, weak reasoning, and CWE errors.
- Prompts are derived on DVJA and retested on the Juliet suite.
- Cross-model validation checks whether the refined prompt generalizes.
From the original article
1 min read at arxiv.orgLarge Language Models have emerged as promising tools for software vulnerability analysis, but their effectiveness depends heavily on prompt design. Existing research primarily compares prompting strategies using aggregate performance metrics, providing limited insight into why models fail or how prompts can be improved systematically.
By Mandana Ghadamian · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.