PEARL durably suppresses misaligned behaviors in language models
Original title: Suppressing Misaligned Behaviors in Language Models, Durably
PEARL augments GRPO rollouts with intermediate state continuations for durable suppression while preserving accuracy.
Two model organisms are constructed to study post-hoc suppression of misaligned behaviors: narrow-trigger and broad-trigger. Standard methods including SFT, GRPO and gradient-ascent unlearning are compared on immediate suppression, durability and capability preservation. PEARL, a GRPO variant with prefix-expanded adversarial reinforcement learning, achieves strong immediate suppression and remains durable under further training on either the organism or unrelated data.
- Evaluates suppression criteria including durability under fine-tuning
- PEARL outperforms on broad-trigger organisms
- Preserves task accuracy across Qwen3-4B and GPT-OSS-20B
- Benefits greatest when misalignment is broad
- Recovers from wider range of misalignment states
From the original article
1 min read at arxiv.orgUndesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior.
By Luxi He · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.