1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
An information-efficiency ratio lets sparse on-policy distillation match full supervision at 0.1%–1% of tokens.
The paper studies noisy gradient estimates when sparse on-policy distillation supervises only a small subset of student-generated tokens. It defines an information-efficiency ratio from a signal-to-noise decomposition and uses a candidate-set approximation to pick tokens while keeping the sampled reverse-KL objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors, and 0.1%–1% token budgets match or exceed full on-policy distillation. Code is available as IER-OPD.
- IER measures gradient-estimation reliability under an optimal scalar baseline.
- Token selection combines IER with existing usefulness scores.
- Sparse budgets of 0.1%–1% match or beat full on-policy distillation.
- Gains shown on mathematical and medical reasoning tasks.
- Code is released as IER-OPD.
From the original article
1 min read at huggingface.co
Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition.
By Huanxin Sheng, Zhiling Ye, Haonan Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.