Scaling Properties of Same-Family On-Policy Distillation
On-policy distillation lets larger students surpass compact RL teachers in an early useful-transfer regime.
The paper measures how on-policy distillation transfers RL-induced reasoning across weak-to-strong, same-base, and strong-to-weak teacher-student pairs from the same model family. Early training shows a useful-transfer regime in which held-out gold score rises approximately linearly with the square root of token-level reverse KL from the student initialization. In every observed weak-to-strong pair the student's peak score exceeded the teacher's. Fitted power laws indicate peak score improves with teacher scale only up to roughly the student scale, and that smaller teachers transfer better at a matched gold score; bootstrapping and the degree of on-policy supervision are also studied.
- Early useful transfer is roughly linear in square-root reverse KL
- In every weak-to-strong pair, the student peak beat the teacher
- Power laws relate peak score and slope to model scale
- Teacher scale helps only up to about the student's scale
- Smaller teachers transfer better at a matched gold score
From the original article
1 min read at huggingface.co
*Reinforcement learning (RL)* can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of *on-policy distillation (OPD)* across *weak-to-strong*, *same-base*, and *strong-to-weak* teacher--student setups.
By Yuntai Bao, Qinfeng Li, Guoqing Jiang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.