Forgeable Confirmation in Automated Computer Security Testing: Deterministic Rules versus AI Judges
Checks that trust attacker-controlled data can forge success verdicts in automated AI security testing.
A study of automated computer-security testing argues that success checks are forgeable when they trust attacker-controlled data. In a four-stage AI-assisted pipeline, nine of fifteen confirmation mechanisms were forgeable; predictions fixed in advance separated all sixteen held-out mechanisms and were 99.9% accurate across 12,203 mechanisms in public scanner templates. Deterministic rules failed when 2% of response content was attacker-controlled, compared with a 50% median for eight open-weight LLM judges. Moving decisive evidence to a channel the attacker cannot write cut reported attack success from 97% to 0%, unless the scanned host itself is the adversary.
- Nine of fifteen confirmation mechanisms in the tested pipeline were forgeable.
- Forgeability depended on whether the check read attacker-controlled data.
- Held-out predictions were exact, and 99.9% accurate on 12,203 templates.
- Rules failed at 2% attacker content; LLM judges at a 50% median.
- An unwritable evidence channel dropped reported success from 97% to 0%.
Full article225 words · extracted from arxiv.org · click to collapse
AI is increasingly used to automate computer security testing, and the tools must decide for themselves whether an attack succeeded. A finding that a deterministic rule confirms by observation is reported as fact, whereas one that an LLM judges exploitable is treated as an opinion. We ask whether the system under test can forge that confirmation. In offline security testing of a four-stage AI-assisted pipeline, nine of its fifteen confirmation mechanisms are forgeable, and forgeability is predicted entirely by whether the decision reads attacker-controlled data. We formalise this as an auditable attack surface and test it prospectively: on sixteen held-out mechanisms, predictions fixed before any attack separated forgeable from unforgeable mechanisms exactly, and across 12,203 mechanisms in public scanner templates the prediction was 99.9% accurate. Deterministic rules proved cheaper to forge than eight open-weight LLM judges, failing at 2% of attacker-controlled response content against a median of 50%. No implementation of one check was both robust and precise, and routing between a rule and an AI judge raised forgery to 99%. Moving the decisive evidence to a channel the attacker cannot write cuts attack success from 97% to 0%, and an escalate verdict recovers the sensitivity this costs. The protection fails when the scanned host is itself the adversary. The results bear on AI security agents and on benchmarks that score success by string matching.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.24200