AOPD weights teacher actions by student advantage in on-policy distillation
Original title: When to Follow the Teacher: Advantage-Based On-Policy Distillation
The estimator needs only student rollouts and terminal rewards, with no extra rollouts or reward evaluations.
The paper argues that standard online policy distillation forces a student to follow the teacher at every state even when that guidance does not help. Advantage-Based On-Policy Distillation reinforces teacher actions with positive counterfactual advantage under the student and downweights the rest. The objective is estimated from ordinary student rollouts and terminal rewards using teacher importance weights, without additional rollouts. Experiments on math, coding, and Medqa report gains over distillation and reinforcement-learning baselines in external-teacher and self-distillation settings.
- Standard OPD assumes teacher guidance is beneficial at every state.
- AOPD uses teacher importance weights on student-sampled actions.
- The estimator avoids trying alternative teacher actions with new rollouts.
- Gains are reported for both external-teacher and self-distillation.
- Tasks covered are math, coding, and Medqa.
From the original article
1 min read at arxiv.orgOnline Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every state of the trajectory, assuming that teacher guidance is always beneficial.
By Mahdi Sabbaghi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.