Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Researchers propose Negative Self-Distillation (NSD), a label-free LLM self-improvement method that diverges from self-generated flawed reasoning rather than imitating privileged solutions.
The authors show On-Policy Self-Distillation can degrade complex reasoning by forcing imitation of artificially confident traces built on privileged information, suppressing uncertainty and self-correction. NSD instead generates a question-specific negative condition — such as acting as a 'careless reasoner' — and pushes the model's distribution away from it without ground-truth labels. A dynamic gating mechanism isolates reasoning-critical tokens so gradient updates fix behavioral flaws without damaging foundational linguistic capabilities. NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning baselines.
- OPSD can degrade complex reasoning by forcing imitation of confident privileged traces.
- Dynamic gating targets gradient updates to reasoning-critical tokens, preserving language ability.
- NSD outperforms OPSD and other label-free self-bootstrapping RL baselines.
From the original article
1 min read at huggingface.co
On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for large language model (LLM) self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions.
By Rongcan Pei, Zhepei Wei, Shuyao Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.