On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
Controlled distillation finds KL direction matters more than on-policy versus off-policy rollouts.
The study isolates rollout policy, token-level KL direction, and learning rate in strong-to-weak distillation on Llama 3 and Qwen2.5 across scientific, medical, and arithmetic reasoning. Token-level KL direction shapes task performance and output coverage more than whether rollouts are on-policy, while learning rate governs forgetting and update sparsity. Forward KL remains robust across rollout policies, whereas reverse KL is more sensitive and favors student-generated rollouts. On-policy data improves harder Countdown variants, but that advantage does not reliably persist after subsequent RLVR.
- Independently varies rollout policy, KL direction, and learning rate.
- Uses Llama 3 and Qwen2.5 on science, medicine, and arithmetic.
- Forward KL stays robust; reverse KL favors student-generated rollouts.
- On-policy Countdown gains do not reliably persist after later RLVR.
From the original article
1 min read at huggingface.co
On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalisation. However, existing comparisons between supervised fine-tuning and reinforcement learning vary many factors simultaneously, making the contribution of rollout policy difficult to isolate.
By Julianna Piskorz, Antonin Berthon, Mihaela van der Schaar · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.