Guardrail safety scores track policy labels more than detection capability
Original title: Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On
Ten arms trained on about 53k dual-labeled prompts show HarmBench-copyright scores spanning 0-100 while detection stays unchanged.
A paper argues that "unsafe" for a guardrail is a property of the labeling policy, not of the text. The same roughly 53k prompts were labeled under two operational definitions and used to train ten guardrail arms at two scales. After decontamination, benchmark spread was driven by label definition: HarmBench-copyright ranged 0-100 and OR-Bench-hard over-refusal 3.4-56.4 percent, while detection capability was unchanged. The authors say evaluations should report contamination, a seed floor, and the graded definition.
Unconfirmed: Findings come from the authors' analysis of one industrial labeling setup; external replication is not reported.
- JailbreakBench-benign was 88 percent and OR-Bench-hard 82 percent present in training pools.
- Score range across arms was 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign.
- Changing 0.03 percent of parameters moved a benchmark across its full range.
- Validation accuracy tracked policy agreement (rho +0.74), not detection (rho +0.34).
From the original article
1 min read at huggingface.co
A guardrail classifier is reported as if unsafe were a property of text. It is not: it is a property of a policy, and the policy enters the model through its labels. We exploit an accident of an industrial deployment in which the same ~53k prompts were labelled twice, under two operational definitions ("is this a request for harmful content?"
By Vahid Jebraeeli, Jonathon Romero, Yijin Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.