Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD
Selective Direct-OPD keeps the top 10% of high-divergence states and beats dense distillation on math benchmarks.
The paper shows Direct On-Policy Distillation can assign unchanged token-level rewards even when Jensen-Shannon and KL divergence between pre-RL and post-RL teacher checkpoints vanish. S^2D-OPD ranks student-sampled states by teacher-reference JSD and applies Direct-OPD supervision only to the top 10% of states per response. Across two teacher pairs and four students from 1.7B to 8B parameters, it beat dense Direct-OPD on AIME and HMMT in seven of eight settings and matched it in the eighth, without extra forward passes.
- Direct-OPD reward can stay fixed while checkpoint divergence vanishes.
- S^2D-OPD masks low-JSD states and keeps the top 10% per response.
- Gains held on AIME and HMMT in seven of eight student settings, from 1.7B to 8B.
- No extra teacher forward passes are required.
From the original article
1 min read at huggingface.co
Direct On-Policy Distillation (Direct-OPD) transfers reinforcement-learning-induced policy improvements from a small model to a larger student by using the token-level log-ratio between post-RL and pre-RL checkpoints as dense supervision on the student's own rollouts.
By Yibo Zhao, Zixuan Yang, Yunshi Lan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.