Study finds LLM jailbreak graders collapse the compliance class
Original title: The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check
Across eight graders, compliance rates on two 900-response corpora differed by a factor of 22 to 25.
A paper argues jailbreak attack-success rates depend on the grader, not only the target model. Holding the model, prompts and responses fixed, three dense 8B and 9B graders moved compliant responses into the neighboring evasion class; one assigned compliance zero times in 900 responses. The collapse also appeared on new harmful requests, a third-family target and a held-out 7B grader. On two corpora, compliance differed by a factor of 22 to 25, while non-refusal differed by 18 to 23 percentage points. The authors propose reporting per-class counts and the named grader.
- Without the three collapsing graders, compliance differed by a factor of 1.7 to 2.8.
- Non-refusal still differed by 10 to 22 points without those graders.
- Results did not change when every grader was called once.
- Per-class label counts expose the collapse without human labels.
- 1.Report the named grader, compliance rate and per-class label counts together.
From the original article
1 min read at arxiv.orgJailbreak papers report an attack success rate, the share of responses that deliver the harmful request, and read it as a property of the target model. We hold the target model, the prompts and the responses fixed and change only the grader. Three dense graders of 8B and 9B parameters, from three model families, move compliant responses to the neighbouring class, evasion, instead of scattering them across classes.
By Ruqiya Bin Safi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.