OPMD distills teacher intermediate computation into on-policy students
Original title: OPMD: On-Policy Mechanism Distillation Beyond Output Supervision
The paper reports the highest average accuracy among distillation baselines in all three teacher-student settings.
On-Policy Mechanism Distillation transfers a teacher's intermediate computation rather than only its outputs. It first builds a student-compatible teacher whose states the student's later layers can use, then internalizes those states from on-policy trajectories. The authors say it also generalizes to multi-teacher distillation and cross-family supervised fine-tuning.
- Standard on-policy distillation supervises only the output space.
- Hidden-state matching can fail when representations differ geometrically.
- Analyses say mechanism supervision changes the student's solution strategies.
- The method is evaluated in three teacher-student settings.
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains confined to the output space. Consequently, OPD transfers what a teacher predicts without supervising the intermediate computation that produces those predictions.
By Yingdong Shi, Zhihong Li, Changming Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.