Paper measures mismatch on-policy distillation can tolerate
Original title: How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
Model-specific FP8 recipes kept quality and fit at least 1.93x as many KV-cache tokens as BF16.
A study measures how quantization, weight staleness, and injected noise create training-inference mismatch in on-policy distillation. The authors report that OPD tolerates substantial mismatch before quality degrades, and stays stable under rollout settings that destabilize reinforcement learning. They add a lightweight admission check for rollout configurations and say FP8 plus asynchronous rollouts reached the same quality 1.62x sooner than synchronous BF16.
Unconfirmed: The mismatch budget and speed claims come from the authors' controlled experiments.
- Quantization was tested across three student-teacher pairs.
- Weight staleness and injected noise were tested on one pair.
- Some layers were particularly sensitive to reduced precision.
- Weight noise shifted generation prefixes beyond per-token correction.
- Code is at github.com/benyucong/train-inference-mismatch-opd.
From the original article
1 min read at huggingface.co
On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained.
By Cong Yu, Ziting Wang, Songlin Jiang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.