Study finds on-policy distillation tolerates training-inference mismatch

Authors report on-policy distillation keeps quality under substantial quantization and rollout mismatch, with faster FP8 training.

A paper studies training-inference mismatch in on-policy distillation, where a student language model is trained on its own responses under a fixed teacher. Quantization, weight staleness, and injected noise were measured; quantization was tested across three student-teacher pairs. The authors report that on-policy distillation tolerates substantial mismatch before quality drops and stays stable under rollout settings that destabilize reinforcement learning. Model-specific FP8 recipes kept quality while holding at least 1.93 times more KV-cache tokens than BF16, and FP8 with asynchronous rollouts reached the same quality 1.62 times sooner than synchronous BF16. They describe a lightweight admission check for rollout configurations; code is published on GitHub.