MTA-OPSD lets each token pick its own teacher in self-distillation
Original title: Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation
On Qwen3 math tasks, the method beats prior on-policy self-distillation, with gains of up to 5.9 over the base model.
The paper argues that on-policy self-distillation should not keep one teacher fixed across a response, because preferred supervision changes token by token. A pilot study uses majority voting and KL-based selection to identify a preferred teacher per token. Multi-teacher Adaptive On-policy Self-Distillation builds a pool of candidate teachers and lets each token choose during training. Code is released.
- Existing OPSD either fixes the teacher or varies references only across responses.
- A pilot study finds teacher preference differs across tokens in one response.
- Experiments use Qwen3 models on mathematical reasoning tasks.
- Reported average gain is up to 5.9 over the base model.
- Code is at github.com/xingzhejun/MTA-OPSD.
From the original article
1 min read at arxiv.orgOn-policy Self-distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses.
By Yifu Luo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.