AWS puts AI vulnerability detection to the test, and false positives pile up
AWS publicly released its Deception Benchmark (14,822 samples) showing leading AI models falsely flag 41-99% of safe code as vulnerable.
AWS released its Deception Benchmark publicly, containing 14,822 samples across 16 programming languages and more than 70 CWE categories, with 9,695 scored samples split into 6,988 code-level and 2,707 environment-gated challenges. AWS evaluated 12 models from five providers using single-turn prompts and found none met its production bar of below 10% for both false-positive and false-negative rates. With direct prompting, models caught nearly all real vulnerabilities but incorrectly flagged 41% to 99% of safe code, with precision between 52% and 71%. Asking models to prove exploitability reduced false positives by 17 to 74 percentage points but raised false-negative rates to 7-44%, with models struggling most when external controls like Kubernetes Network Policies blocked apparent exploits.
- Benchmark spans 14,822 samples, 16 languages, 70+ CWE categories; 9,695 are scored
- No tested model met AWS's sub-10% false positive and false negative production bar
- Direct prompting produced 41-99% false positive rates on safe code
- Proof-of-exploit prompting cut false positives 17-74 points but raised misses to 7-44%
- Labels withheld; users submit predictions to AWS for verified scoring
From the original article
3 min read at helpnetsecurity.com
AWS’ Deception Benchmark measures how well AI models distinguish genuine security vulnerabilities from code that looks risky but is safe. AWS is making it publicly available so researchers can use the dataset and evaluation process without repeating the cost of generating and refining the samples. Security teams use AI for vulnerability triage, penetration testing, threat modeling, incident response, and code review.
By Anamarija Pogorelec · helpnetsecurity.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.