Multi-teacher adaptive on-policy self-distillation
Original title: Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation
Token-level adaptive self-distillation improves math reasoning on Qwen3 models.
The method creates a multi-teacher pool and allows every token to select its preferred teacher during training. It outperforms existing OPSD approaches with average gains up to +5.9. Code released at GitHub repository.
- Every token gets its own teacher
- Identified via majority voting and KL selection
- Average gain of +5.9 over base model
- Available at https://github.com/xingzhejun/MTA-OPSD
- AI models
- Qwen3
From the original article
1 min read at huggingface.co
On-policy Self-distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses.
By Yifu Luo, Chuang Ma, Zeyu Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.