Rethinking Relative Policy Differences Transfer of On-Policy Distillation
Relative policy differences, not expert strength alone, determine useful on-policy distillation transfer.
The paper treats the log-probability ratio between a proxy expert and a proxy reference as an implicit reward for KL-regularized policy optimization. This induces an on-policy update, recovers conventional on-policy distillation, and supports transfer across models differing in scale, training stage, or generation. Experiments find task-specific reinforcement learning and supervised fine-tuning differences transfer consistently, while differences from model scale alone transfer weakly. A suitable reference can isolate the relevant residual and rescue otherwise unsuccessful transfer.
- Relative expert-reference log-probability ratios act as an implicit reward for on-policy distillation.
- RL and supervised fine-tuning differences transfer; scale-only differences transfer weakly.
- The expert supplies capability while the reference filters shared behavior.
- Raw policy distance does not predict successful transfer.
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target. Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution.
By Rong Wu, Daocheng Fu, Yu Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.