Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
Cross-tokenizer on-policy distillation works better with strict token alignment than broader, less reliable supervision.
The paper studies on-policy distillation when teacher and student use different tokenizers, testing whether wider alignment coverage improves learning. Across three heterogeneous pairs on mathematical reasoning and code generation, strict 1:1 groups already cover most student tokens, and the shared vocabulary retains nearly all probability mass at aligned positions. Restricting reverse KL to a student-selected top-16 of the shared vocabulary matches full shared-vocabulary OPD and beats evaluated baselines, while mean-squared-error supervision on mismatch spans lowers accuracy because those gradients conflict with the strict loss.
- Strict 1:1 alignment already covers most student-generated tokens.
- Top-16 shared-vocabulary reverse KL matches full shared-vocabulary OPD accuracy.
- Adding span MSE supervision covers mismatches but reduces accuracy.
- Span gradients weakly or negatively agree with strict-loss gradients.
From the original article
1 min read at huggingface.co
On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this paper, we examine whether expanding this alignment coverage improves learning.
By Bingxi Hou, Guochao Jiang, Guofeng Quan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.