P^2-MOPD accelerates multi-teacher distillation with parameter priors
Original title: P^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors
P^2-MOPD jointly optimizes mixing coefficients and student weights to reach 94.7% recovery in half the steps of MOPD.
Multi-teacher on-policy distillation supervises student trajectories from expert models. Task arithmetic merges parameters linearly but stays within frozen spans. P^2-MOPD treats the combination as a differentiable model and optimizes coefficients plus student weights together. Experiments across domains at 1.7B and 4B scales show faster balanced recovery.
- Keeps student trainable while using experts as priors
- Achieves 94.7% average recovery at 1.7B scale
- Surpasses MOPD final recovery in half training steps
- Joint optimization of coefficients and weights
- Faster balanced recovery across four domains
From the original article
1 min read at huggingface.co
Multi-teacher on-policy distillation consolidates expert capabilities by supervising student-generated trajectories, whereas task arithmetic offers a training-free alternative through linear parameter combinations.
By Sinuo Wang, Jiacheng Lu, Zeyu Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.