Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
OASIS keeps on-policy self-distillation effective at scale by supervising verified reasoning trajectories.
The paper analyzes on-policy self-distillation and finds scaffold correctness matters more for downstream accuracy than teacher-context correctness. Unverified scaffolds create an imitation gap that shrinks with scale, while standard OPSD still mostly supervises unverified trajectories. OASIS keeps the OPSD objective but supervises mostly label-verified on-policy trajectories and uses unverified model attempts as teacher context, needing only final-answer labels. On Qwen3-1.7B, 4B, and 8B, OASIS gains 3.2 to 3.8 points on average across AIME 2024, AIME 2025, and HMMT 2025, and beats OPSD by 3.05 points at 8B.
- Scaffold correctness outweighs privileged teacher context.
- OASIS needs only final-answer labels, not written solutions.
- Average gain is 3.2 to 3.8 points over base models.
- At 8B, OASIS beats OPSD by 3.05 points.
Full article205 words · extracted from huggingface.co · click to collapse
On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solution. We separate these roles in a factorial analysis and find that scaffold correctness has a stronger effect on downstream accuracy than context correctness. Unverified scaffolds create an imitation gap because the teacher can use information unavailable to the student. This gap shrinks with model scale, yet OPSD continues to supervise mostly unverified trajectories. In contrast, verified scaffolds remain effective even when the teacher is conditioned on the student's own unsuccessful rollout. Based on this finding, we introduce OASIS, which retains the OPSD objective but supervises mostly verified by label on-policy trajectories and replaces written solutions with unverified model-generated attempts as the teacher context. OASIS therefore requires only final-answer labels. Across Qwen3-1.7B, 4B, and 8B on AIME 2024, AIME 2025, and HMMT 2025, OASIS improves over the base model by 3.2--3.8 points on average, while OPSD's gain falls from 3.05 points at 1.7B to 0.14 at 8B. At 8B, OASIS improves over OPSD by 3.05 points, showing that verified on-policy scaffolds preserve the effectiveness of self-distillation as models scale.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.37915