Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts
Δ-MOPD distills multiple teachers via teacher-relative logit shifts instead of endpoint policies, beating endpoint composition by 4.11 Math points.