Paper proposes ASR_STEP to discount iterative LLM jailbreaks
Original title: Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
On JailBreakBench, discounting victim-model calls lowers GCG by 7.0 to 8.7 points and PAIR by 3.4 to 12.
A paper argues that attack success rate ignores repeated queries that give a deployed system more chances to detect a jailbreak. ASR_STEP keeps binary success judgments while penalizing additional calls to the victim model. The evaluation covers GCG, PAIR with a step budget, and a single-step PAP baseline on JailBreakBench across three model families. PAP is unchanged because it uses one step, while the iterative methods are discounted. The authors publish their code.
- GCG scores fall by 7.0 to 8.7 percentage points after discounting.
- PAIR's discount ranges from 3.4 to 12 percentage points.
- Single-step PAP remains unchanged under ASR_STEP.
- The authors recommend reporting both ASR and ASR_STEP.
From the original article
1 min read at arxiv.orgAttack success rate (ASR) evaluates jailbreaks against large language models without accounting for repeated calls to the victim model. Yet each additional call gives the deployed system another opportunity to detect the attack and intervene. We introduce $\mathrm{ASR}_{\mathrm{STEP}}$, which penalizes additional victim-model calls while retaining binary success judgments.
By Rohan Menon · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.