Jailbreak detectors need low false-positive tests, not AUROC alone
Original title: Beyond AUROC: Deployment-Aware Evaluation of Model-Internal Jailbreak Detectors
Thresholds set on curated benign prompts often failed to hold false-positive budgets on real-user traffic.
The paper evaluates model-internal jailbreak detectors at low false-positive operating points rather than by AUROC alone. Across four detectors, four LLMs, and twelve jailbreak families, calibration on curated benign prompts often failed to preserve false-positive budgets on real-user traffic. Recalibration improved control but exposed family-level coverage gaps, including high AUROC with low true-positive rate. Changing reference-set composition shifted which attack families were detected.
- Improving coverage of one attack family can reduce coverage of others.
- Authors treat calibration and reference data as part of detector security behavior.
- They recommend per-family true-positive rate at explicit low false-positive points.
- AUROC is recommended only as a complementary ranking metric.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) are increasingly deployed in user-facing systems, where jailbreak attacks can bypass safety controls and induce harmful and policy-violating outputs.
By Mohammad Tehabsim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.