Obfuscation leads jailbreak success across seven evaluated LLMs
Original title: Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models
Obfuscation averaged a 54.3% attack success rate, and self-reminder prompting cut ASR to 10% and 5% on two models.
A study tests five jailbreak families—direct requests, roleplay, hypothetical framing, instruction override, and obfuscation—on seven open-weight and API-accessible LLMs. Obfuscation was the most transferable vector and bypassed every model with nonzero success. Model size did not predict robustness, and models in the same family differed sharply. Four defenses were tested on the two most vulnerable models; self-reminder prompting was the most consistent and added no inference cost or false positives. Code and datasets are public.
Unconfirmed: Attack success rates are the authors' measurements on their public dataset, not reports of production incidents.
- Models tested: Llama-3-8B, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B, Qwen2.5-7B, Gemma2-9B, and Phi-4-14B.
- Size and robustness had Spearman r=-0.57, p=0.18.
- Self-reminder reduced ASR from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B.
- Other defenses were keyword blacklisting, paraphrase sanitization, and perplexity filtering.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) deployed in safety-critical settings must resist adversarial inputs designed to elicit harmful outputs. We present a systematic empirical evaluation of five adversarial jailbreaking attack families---direct requests, roleplay-based jailbreaking, hypothetical framing, instruction override, and obfuscation---across seven open-weight and API-accessible LLMs: Llama-3-8B, Llama-3.
By Md. Robiul Islam Niloy · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.