Paper measures how much mismatch on-policy distillation tolerates
Original title: How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
Authors say model-specific FP8 recipes keep quality while holding at least 1.93 times more KV-cache tokens than BF16.
A paper studies training-inference mismatch in on-policy distillation, where a student language model is trained on its own responses under a fixed teacher. Quantized inference and asynchronous rollouts cut generation cost but can diverge from the model being trained. Across three student-teacher pairs, the authors report that OPD tolerates substantial mismatch before quality drops, and stays stable under rollout settings that destabilize reinforcement learning. They also describe a lightweight admission check and say FP8 plus asynchronous rollouts reaches the same quality 1.62 times sooner than synchronous BF16.
Unconfirmed: Tolerance figures and speedups are reported by the authors only.
- Quantization, weight staleness, and injected noise are measured on one scale.
- Some layers are described as particularly sensitive to reduced precision.
- Weight noise shifts generated prefixes that per-token corrections cannot fix.
- Code is at github.com/benyucong/train-inference-mismatch-opd.
From the original article
1 min read at arxiv.orgOn-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained.
By Cong Yu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.