LLMs overestimate CVSS severity on bug bounty reports
Original title: Narratives Mislead: An Empirical Study of How LLMs Misjudge CVSS Severity in Bug Bounty Reports
Best prompting reached 49.58% level accuracy on 240 vendor-validated Huntr reports; fine-tuning a 7B model hit 56.25%.
Researchers mined 4,281 public reports from Huntr and HackerOne and evaluated eight LLMs on 240 Huntr reports with vendor-validated CVSS vectors. Models overestimated severity far more often than they underestimated and almost never recognized low-severity cases. Errors tracked three boundaries narrative reports blur: potential versus realized impact, magnitude versus authorization scope, and uncertainty versus worst case. A four-report case study did not settle metrics the reports left open.
Unconfirmed: The exploratory case study covers four reports, and two published labels disagreed on three.
- Best prompting was ten points above always predicting the majority level.
- Fine-tuning a 7B model reached at most 56.25%, called an optimistic upper bound.
- Four prompting strategies were compared, plus fine-tuning.
- Published labels disagreed on three of four case-study reports.
From the original article
1 min read at arxiv.orgBug bounty platforms receive vulnerability reports faster than vendors can triage them, and Large Language Models (LLMs) are increasingly considered for assigning the Common Vulnerability Scoring System (CVSS) severity that drives prioritization. Prior evaluations score short CVE descriptions and report accuracy, but deployment depends on how models err, not only how often.
By Ruitao Feng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.