On-Policy Distillation with Negative-Policy Rollouts
NP-OPD improves on-policy distillation by using weaker-model rollouts as an explicit negative signal.
Negative-Policy On-Policy Distillation (NP-OPD) augments teacher-supervised on-policy distillation with rollouts from a weaker negative policy. When the teacher and student have limited distributional overlap, positive teacher imitation alone can give weak learning signals; NP-OPD keeps exposing tokens preferred by the negative policy to teacher supervision without changing the distillation reward. Experiments show gains across model scales, generation modes, reasoning domains, and existing OPD variants. Analyses indicate the method suppresses tokens the negative policy prefers over the teacher and shifts the student away from that policy. Code is planned at the Naver AI repository.
- NP-OPD adds rollouts from a weaker negative policy to standard on-policy distillation.
- Negative-policy tokens stay under teacher supervision instead of changing the reward formula.
- Gains hold across model scales, generation modes, reasoning domains, and OPD variants.
- Analysis shows the student suppresses negative-policy tokens and moves away from that policy.
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher.
By Jaehui Hwang, Dongyoon Han, Sangdoo Yun · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.