When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
Researchers show EOS termination-token mismatch between base students and post-trained teachers drives length inflation in on-policy distillation across Qwen3, Llama, and Gemma.
The paper studies length inflation in on-policy distillation (OPD), where student responses become excessively long and can exhaust the generation budget. The authors identify termination-token mismatch between base students and post-trained teachers as an important cause, observing that Qwen3, Llama, and Gemma place stopping probability on different EOS tokens even with identical declared stopping sets. Treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigates mismatch-induced inflation across all three families, while aligning decoding stopping sets alone is insufficient. A stage-wise analysis of K2-Horizon training shows termination preferences shift during training, and an implementation with the proposed corrections is released.
- Termination-token mismatch between base students and post-trained teachers causes excessive response lengths in on-policy distillation.
- Mismatch found across Qwen3, Llama, and Gemma despite identical declared stopping sets.
- Treating functionally equivalent EOS tokens as one semantic stopping action substantially mitigates length inflation.
- Implementation with termination-handling corrections is released.
Full article183 words · extracted from huggingface.co · click to collapse
We study length inflation in on-policy distillation (OPD), where student responses can become excessively long and even exhaust the generation budget. We identify termination-token mismatch between base students and post-trained teachers as an important source of this behavior. Across Qwen3, Llama, and Gemma, the two models can place their stopping probability on different EOS tokens, even when their declared stopping sets are identical. This mismatch can suppress the student's preferred termination action without reliably transferring the teacher-preferred alternative. We show that aligning the decoding stopping set alone is insufficient, while treating functionally equivalent EOS tokens as a shared semantic stopping action substantially mitigates mismatch-induced length inflation across all three model families. To further understand how termination behavior evolves over training, we study OPD across different K2-Horizon training stages. This stage-wise analysis shows that termination preferences can shift substantially during training, while also revealing a distinct length inflation late in the OPD run that persists beyond termination alignment. Together, these results identify termination mismatch as an important, but not exhaustive, source of OPD length dynamics. We release an implementation incorporating the proposed termination-handling corrections.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.20511