Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Waypoint-OPD gates RL policy-shift distillation and leads math and code scores for Qwen3 students.
Waypoint-OPD transfers reinforcement-learning policy shifts by comparing early and late teacher checkpoints and suppressing the endpoint shift when those shifts disagree. Using RL-trained Qwen3-4B teachers, the method is tested with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B students on mathematical and code reasoning. It records the highest domain-average score among compared distillation methods in all six domain-student settings. Ablations on Qwen3-8B show that removing or shuffling the gates lowers math performance.
- Waypoint-OPD gates endpoint policy shifts by early-versus-late directional agreement.
- It needs one extra frozen-teacher scoring pass versus endpoint-only transfer.
- RL-trained Qwen3-4B teachers distill into 8B, 14B, and 30B-A3B students.
- It led domain-average scores in all six math and code student settings.
- Removing or shuffling gates hurt Qwen3-8B math reasoning.
From the original article
1 min read at huggingface.co
Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other.
By Bingxi Hou, Guochao Jiang, Guofeng Quan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.