Calibrating Teacher--Student Discrepancy for On-Policy Distillation
Researchers propose Cal-OPD, a calibrated on-policy distillation method that filters teacher self-deviation and improves reasoning model training.
Calibrated On-Policy Distillation (Cal-OPD) estimates a teacher model's self-deviation region using positive and negative privileged interventions, then retains only the teacher-student discrepancy beyond that region as the training signal. On mathematical reasoning benchmarks, it consistently outperforms standard on-policy distillation across model scales while using only 52-65% of the original discrepancy signal.
- Cal-OPD separates teacher self-deviation from true teacher-student capability gap
- Uses privileged interventions to estimate teacher deviation regions
- Retains 52-65% of original discrepancy as optimization signal
- Outperforms standard OPD on mathematical reasoning benchmarks
Full article155 words · extracted from huggingface.co · click to collapse
On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce Calibrated On-Policy Distillation (Cal-OPD), which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65\% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.21619