The Model Proposes, the Code Disposes: A Pre-Registered Ablation of a Verifier-and-Acceptance Stage in an LLM-Orchestrated Offensive-Security Agent
Pre-registered ablation finds a model verifier stage in an LLM offensive-security agent suppresses findings; removing it eliminated suppression with precision tradeoff.
The paper evaluates a verifier-and-acceptance stage in an LLM-orchestrated offensive-security agent via a pre-registered 20-run confirmatory ablation and a 2x2 factorial study with 40 runs on vulnerable lab targets. Removing the stage eliminated pre-report suppression (median 2 vs 0 findings, p = 0.00003) but reduced model-blinded shipped precision (0.471 vs 0.353, p = 0.0087). Suppression was attributed to the model verifier rather than deterministic acceptance rules, and an instrumented canary recorded zero external contacts in all 60 runs. The full design retained 93.8% of model-adjudicated true candidates but failed its pre-registered non-inferiority floor of 0.90.
- Removing verifier stage eliminated pre-report suppression (p = 0.00003)
- Suppression attributed to model verifier, not deterministic acceptance code
- Canary recorded zero external contacts in all 60 runs
- Non-inferiority criterion not met: lower 95% bound 0.875 below 0.90 floor
Full article254 words · extracted from arxiv.org · click to collapse
We evaluate whether a verifier-and-acceptance stage - a model verifier whose verdicts are enforced by deterministic code - changes what an LLM-driven offensive-security agent reports. We report a 15-run exploratory pilot, a pre-registered 20-run confirmatory ablation, and a pre-registered 2 x 2 factorial study with 40 runs across two deliberately vulnerable lab targets. In the confirmatory study, removing the stage eliminated pre-report suppression (median 2 versus 0 findings per run; exact one-sided p = 0.00003) and reduced model-blinded shipped precision (median 0.471 versus 0.353; p = 0.0087). Recall against a frozen but incomplete ground-truth list did not differ significantly (two-sided p = 0.158; equivalence was not established). The factorial study attributed suppression to the model verifier (Holm-adjusted p = 0.004); deterministic acceptance rules alone suppressed no false positives, and no interaction was detected (p = 0.72). The full design retained 93.8% of model-adjudicated true candidates but did not meet its pre-registered non-inferiority criterion because the lower one-sided 95% bound was 0.875, below the 0.90 floor. Across the confirmatory and factorial studies, an instrumented canary recorded zero contacts in 60 of 60 runs, with incidental external contacts disclosed separately. Independent human adjudication of the retained blind packets is pending, so precision and sensitivity endpoints are supporting rather than final evidence. Six audit-trail failures, including one in the evaluation tooling, are also disclosed. The results support a narrow conclusion: the verifier changes what the system ships, while deterministic code supplies enforcement and auditability; they do not establish superiority to other agents or generalization beyond lab targets.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.15887