Case-Level Verification in Scanner-LLM Cascades: Overcoming the Alert Aggregation Bottleneck to Expand the FRR-TPR Trade-off Space
Case-level LLM checks of DAST alerts improve false-positive reduction past the alert-aggregation bottleneck.
The paper studies two-stage cascades that use a DAST scanner for detection and an LLM only for verification. Alert aggregation mixes true and false cases in one decision, so removing a false positive can also drop true positives. Standalone LLMs reach 95.4-100% recall but 49.6-85.0% false-positive rates. On DVWA and WebGoat, the best alert-level operating point is FRR 42.86% at TPR 51.7%; case-level verification reaches FRR 47.6%, and a case-focused evidence prompt lifts TPR from 55.2% to 62.1% at that FRR.
- Alert aggregation creates a structural FRR-TPR trade-off bottleneck.
- Standalone LLMs recall 95.4-100% but false-positive rates reach 49.6-85%.
- Case-level verification judges each replayed HTTP finding independently.
- Best alert-level point is FRR 42.86% at TPR 51.7%.
- CEV-Prompt raises TPR from 55.2% to 62.1% at FRR 47.6%.
Full article236 words · extracted from arxiv.org · click to collapse
Dynamic Application Security Testing (DAST) scanners achieve high recall but also produce a large number of false positives, resulting in substantial manual triage costs. Large Language Models (LLMs), when used for independent detection, achieve extremely high recall (95.4%-100%) but also exhibit prohibitively high false positive rates (49.6%-85.0%), precluding their use as standalone replacements for scanners. A natural solution is a two-stage cascade consisting of scanner detection followed by LLM verification. However, a verification-granularity issue that has long been overlooked in practice creates a structural bottleneck: alert aggregation binds multiple true and false cases into a shared decision unit, such that removing a false positive inevitably eliminates true positives aggregated within the same alert group. This creates a trade-off bottleneck between the False-positive Reduction Rate (FRR) and the True-positive Rate (TPR). We formalize this bottleneck by showing that the alert-level false-positive set is a subset of the case-level false-positive set, and introduce a Case-Level, per-case verification strategy that shifts the decision granularity from the alert level to the instance level, independently replaying HTTP requests and making an independent determination for each detected case. Evaluation on the dual testbeds of Damn Vulnerable Web Application (DVWA) and WebGoat shows that the empirically best Alert-Level operating point achieves FRR=42.86% (TPR=51.7%). The Case-Level Baseline achieves FRR=47.6%, an improvement of 4.7 percentage points (+4.7 pp), while the Case-Focused Evidence Verification Prompt (CEV-Prompt) increases TPR from 55.2% to 62.1% at the same FRR.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.08406