Calibrating Teacher--Student Discrepancy for On-Policy Distillation
Researchers propose Cal-OPD, a calibrated on-policy distillation method that filters teacher self-deviation and improves reasoning model training.
Calibrated On-Policy Distillation (Cal-OPD) estimates a teacher model's self-deviation region using positive and negative privileged interventions, then retains only the teacher-student discrepancy beyond that region as the training signal. On mathematical reasoning benchmarks, it consistently outperforms standard on-policy distillation across model scales while using only 52-65% of the original discrepancy signal.
- Cal-OPD separates teacher self-deviation from true teacher-student capability gap
- Uses privileged interventions to estimate teacher deviation regions
- Retains 52-65% of original discrepancy as optimization signal
- Outperforms standard OPD on mathematical reasoning benchmarks
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student.
By Qiangqiang He, Jin Li, MingCai Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.