What Does Privileged Information Add to On-Policy Self-Distillation?
Study of on-policy self-distillation on 5,319 AMPLE-Math problems finds reference-free distillation drives most gains, with modest extra benefit from privileged references.
The authors build AMPLE-Math, a suite of 5,319 mathematical problems with six reasoning views sharing the same answers, to isolate the value of privileged references in on-policy self-distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for most of Qwen3-1.7B's improvement in domain and on external benchmarks; extra reference benefit is modest, strongest for polished solutions, while complete traces add two percentage points for SmolLM3-3B at step 50. Gains reverse when short direct-response rollouts are replaced with long thinking-enabled rollouts. The findings suggest OPSD improves access to existing reasoning capabilities via cross-mode parameter sharing, with references valued for what they add to that transfer.
- Introduces AMPLE-Math: 5,319 math problems with six reasoning views sharing the same answers.
- Reference-free distillation accounts for most of Qwen3-1.7B's improvement in domain and external benchmarks.
- Privileged-reference benefits are modest in Qwen; complete traces add two points for SmolLM3-3B.
- Gains reverse when short direct-response rollouts are replaced with long thinking-enabled rollouts.
Full article226 words · extracted from huggingface.co · click to collapse
On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, and compare each view with matched reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for much of Qwen3-1.7B's improvement under thinking-enabled evaluation, both in domain and on external benchmarks. Evidence for an additional reference benefit is modest in Qwen, strongest for a polished solution, whereas complete traces add two percentage points in SmolLM3-3B at step 50. These benefits depend on the student being trained. At the same checkpoint, replacing short direct-response rollouts with long thinking-enabled rollouts turns gains into losses in both families while the problems, references, and evaluation stay fixed. Teacher profiles and matched loss interventions in Qwen further show that changing token-level supervision can leave student behavior largely unchanged. Together, these findings suggest that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference. The value of a privileged reference is what it adds to this cross-mode transfer, not how much of the solution it reveals.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.20612