Order-Only On-Policy Distillation improves LLM students
Original title: Order-Only On-Policy Distillation
O^{3}PD trains students to match teacher token ordering without probability values or temperature tuning.
On-policy distillation trains students on teacher token probabilities. These entangle preferred tokens and However, O^{3}PD uses only the order of top-K tokens. The method is invariant to temperature. Across six teacher-student pairs, O^{3}PD outperforms standard on-policy distillation on HMMT25, AIME24, and AIME25 by up to 2.9 points.
- Reproduces top-K token order
- Invariant to teacher temperature
- Outperforms OPD by up to 2.9 points
- Gains up to 5.6 on AIME25
- Six teacher-student pairs
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) trains a student on its own generations by matching the teacher's token probabilities. These probabilities entangle two kinds of information: which tokens the teacher prefers, and how confident it is. The latter reflects the teacher's own capacity and calibration, which a smaller student may be unable to reproduce.
By Guangzeng Han, Yu Wu, Xiaolei Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.