When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Semi-OPD, distilling on offline student rollouts, beats on-policy distillation in 14 of 17 teacher-student pairs.
The paper asks whether on-policy distillation is always better for transferring teacher capabilities to student models. Semi-OPD, which distills from offline rollouts of the initial student, outperforms on-policy distillation in 14 of 17 teacher-student pairs ranging from 1.5B to 235B parameters, with up to 13.6% higher accuracy and 11.4x training speedup. On-policy distillation helps only when teacher and student are highly aligned, as measured by output-token overlap. For misaligned pairs, student rollouts become increasingly off-policy relative to the teacher as context length grows, while Semi-OPD remains more stable on shorter contexts.
- Semi-OPD beats OPD on 14 of 17 pairs from 1.5B to 235B
- Peak gains are 13.6% accuracy and 11.4x training speedup
- OPD helps only when teacher and student token overlap is high
- Longer contexts make misaligned student rollouts off-policy to the teacher
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs?
By Siyan Zhao, Yonggan Fu, Jindong Jiang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.