Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Researchers propose Negative Self-Distillation (NSD), a label-free LLM self-improvement method that diverges from self-generated flawed reasoning rather than imitating privileged solutions.
The authors show On-Policy Self-Distillation can degrade complex reasoning by forcing imitation of artificially confident traces built on privileged information, suppressing uncertainty and self-correction. NSD instead generates a question-specific negative condition — such as acting as a 'careless reasoner' — and pushes the model's distribution away from it without ground-truth labels. A dynamic gating mechanism isolates reasoning-critical tokens so gradient updates fix behavioral flaws without damaging foundational linguistic capabilities. NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning baselines.