On the Off-Policy Teacher in On-Policy Distillation
SCOUT adapts teachers to student-generated prefixes and improves on-policy distillation across scales and domains.
On-policy distillation trains a student on its own trajectories under dense teacher supervision, but those prefixes are off-policy for the teacher. The authors find the teacher's continuation performance degrades as student-generated prefixes grow longer. SCOUT periodically updates the teacher with reinforcement learning and verifiable rewards on continuations from student prefixes. Across teacher-student configurations, model scales, and reasoning domains, SCOUT improves distillation effectiveness.
- Teacher continuation quality falls as student prefixes lengthen
- SCOUT adapts the teacher with reinforcement learning and verifiable rewards
- Teacher continuation from student prefixes improves in controlled tests
- Distillation gains hold across scales and reasoning domains
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher.
By Langlin Huang, Hao Liu, Mononito Goswami · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.