Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
On-policy distillation can boost language models or collapse them into repetitive reward hacking.
Researchers explain divergent outcomes of on-policy distillation (OPD) for language-model post-training from a reinforcement-learning view. The teacher implicitly rewards student behaviors, including ones it rarely produces itself. When that reward is reliable, OPD makes correct responses easier to sample without expanding capabilities; when preferences misalign with quality, it amplifies overlong, repetitive rollouts. Masking unhealthy responses during training and using SFT initialization each mitigate the collapse. Code is available at github.com/HancCui/opd_hacking.
- OPD improves sampling of correct answers without expanding student capabilities.
- Unreliable implicit teacher rewards amplify long, repetitive student rollouts.
- Masking unhealthy responses and SFT initialization each mitigate collapse.
- Reliability of the teacher's evaluation matters more than its own generation quality.
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood.
By Han Cui, Jianhao Yan, Yun Luo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.