DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation
DiffGate gates bounded teacher guidance to failed rollouts, lifting Qwen3 pass rates over matched GRPO.
DiffGate is an outcome-gated on-policy distillation objective that combines Group Relative Policy Optimization with selective, bounded teacher guidance. Teacher supervision is used only on failed trajectories, scaled by group difficulty, and limited so extreme teacher-student discrepancies do not dominate optimization. On Qwen3-0.6B and Qwen3-1.7B, code avg@8 rises by 1.7 and 1.8 points and pass@8 by 1.6 and 5.7 points versus matched GRPO. Mathematics avg@8 stays within 0.5 points of GRPO while pass@8 improves by 1.1 and 3.9 points.
- Teacher guidance is applied only to failed trajectories and scaled by group difficulty.
- Guidance is smoothly bounded so large teacher-student gaps do not dominate updates.
- On code, Qwen3-0.6B and 1.7B gain +1.7 and +1.8 avg@8 over matched GRPO.
- Pass@8 improves in all four model-domain settings, including mathematics.
Full article235 words · extracted from huggingface.co · click to collapse
On-policy distillation (OPD) has emerged as a widely used paradigm for post-training large language models, reducing the train--test mismatch of conventional distillation by supervising the student on its own generated trajectories. However, existing OPD objectives remain largely token-local and outcome-agnostic, optimizing teacher--student agreement at each prefix despite reasoning quality being determined at the trajectory level. Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), provides complementary outcome-level supervision but suffers from sparse rewards and coarse credit assignment. We show that OPD and RLVR exhibit complementary blind spots: teacher signals provide dense local guidance but are weakly aligned with rollout correctness, whereas group-relative rewards capture task success but provide coarse token-level credit and vanish on all-failure groups. We introduce DiffGate, an outcome-gated objective that combines GRPO with selective, bounded teacher guidance. Teacher supervision is applied only to failed trajectories, scaled by group difficulty, and smoothly bounded to prevent extreme teacher--student discrepancies from dominating optimization. The verifier therefore determines which trajectories receive teacher guidance, while the teacher provides dense token-level update directions within those trajectories. Across Qwen3-0.6B and Qwen3-1.7B students, DiffGate improves code avg@8 over matched GRPO by +1.7 and +1.8 points and pass@8 by +1.6 and +5.7 points, respectively. On mathematics, avg@8 remains within 0.5 points of GRPO while pass@8 improves by +1.1 and +3.9 points. Overall, DiffGate improves pass@8 across all four model--domain settings, demonstrating improved solution coverage under our evaluation protocol.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.04596