Forgeable Confirmation in Automated Computer Security Testing: Deterministic Rules versus AI Judges
Checks that trust attacker-controlled data can forge success verdicts in automated AI security testing.
A study of automated computer-security testing argues that success checks are forgeable when they trust attacker-controlled data. In a four-stage AI-assisted pipeline, nine of fifteen confirmation mechanisms were forgeable; predictions fixed in advance separated all sixteen held-out mechanisms and were 99.9% accurate across 12,203 mechanisms in public scanner templates. Deterministic rules failed when 2% of response content was attacker-controlled, compared with a 50% median for eight open-weight LLM judges. Moving decisive evidence to a channel the attacker cannot write cut reported attack success from 97% to 0%, unless the scanned host itself is the adversary.
- Nine of fifteen confirmation mechanisms in the tested pipeline were forgeable.
- Forgeability depended on whether the check read attacker-controlled data.
- Held-out predictions were exact, and 99.9% accurate on 12,203 templates.
- Rules failed at 2% attacker content; LLM judges at a 50% median.
- An unwritable evidence channel dropped reported success from 97% to 0%.
From the original article
1 min read at arxiv.orgAI is increasingly used to automate computer security testing, and the tools must decide for themselves whether an attack succeeded. A finding that a deterministic rule confirms by observation is reported as fact, whereas one that an LLM judges exploitable is treated as an opinion. We ask whether the system under test can forge that confirmation.
By Akihisha Fujiyama · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.