LastOPD: Taming Collapse in Latent On-Policy Distillation
LastOPD briefly aligns only the last-layer latent state when distilling Qwen3 into Qwen3-1.7B, beating token-only OPD on MATH-500.
Full latent on-policy distillation of Qwen3-4B and Qwen3-8B into Qwen3-1.7B-Base raised MATH-500 from 25 to 46 in 10 steps, then collapsed to 11 even as the alignment metric improved. The authors argue depth-paired layers play different roles, so continued alignment pulls the student toward teacher states it cannot use. LastOPD applies the latent signal only at the last-layer state during a 10-step crossfade into token-level OPD. It beats token-only OPD by 5.55 and 4.02 MATH-500 points with the 4B and 8B teachers and reaches that baseline's final score in about half the steps.
- Latent supervision lifted MATH-500 from 25 to 46, then collapsed it to 11.
- Alignment kept improving while the most aligned student performed worst.
- LastOPD applies latent loss only at the last layer for a 10-step crossfade.
- MATH-500 rose 5.55 and 4.02 points over token-only OPD with 4B and 8B teachers.
- AI models
- Qwen3-4BQwen3-8BQwen3-1.7B-Base
From the original article
1 min read at huggingface.coOn-policy distillation (OPD) corrects a student on the responses it writes, but its signal is the teacher's next-token distribution: it tells the student what the teacher says but misses how it thinks. Latent supervision promises the missing part by aligning the student's latent states to the teacher's. Recent methods such as OPRD bring this signal into on-policy distillation.
By Jie Yang, Zhengyu Fang, Zelin Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.