1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation
An information-efficiency ratio lets sparse on-policy distillation match full supervision at 0.1%–1% of tokens.
The paper studies noisy gradient estimates when sparse on-policy distillation supervises only a small subset of student-generated tokens. It defines an information-efficiency ratio from a signal-to-noise decomposition and uses a candidate-set approximation to pick tokens while keeping the sampled reverse-KL objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors, and 0.1%–1% token budgets match or exceed full on-policy distillation. Code is available as IER-OPD.
- IER measures gradient-estimation reliability under an optimal scalar baseline.
- Token selection combines IER with existing usefulness scores.
- Sparse budgets of 0.1%–1% match or beat full on-policy distillation.
- Gains shown on mathematical and medical reasoning tasks.
- Code is released as IER-OPD.
Full article143 words · extracted from huggingface.co · click to collapse
Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition. IER characterizes relative gradient estimation error under an optimal scalar baseline. A candidate-set approximation enables token selection based on IER and its combination with existing usefulness scores, while retaining the sampled reverse-KL training objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors in multiple settings, with sparse configurations matching or exceeding full OPD without token selection at small token budgets of 0.1\%--1\%. These results support accounting for both usefulness and gradient-estimation reliability when allocating sparse supervision. Our code is available at https://github.com/BruceSheng1202/IER-OPD.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.24432