When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment
UECR-GRPO blends teacher and verifier credit, slightly improving Qwen3 math-reasoning accuracy.
The paper introduces UECR-GRPO, which combines verifier rewards and teacher guidance inside a single GRPO-style update for mathematical reasoning. Path-Utility Unification mixes a length-normalized teacher score with the verifier reward before group normalization and PPO clipping. Entropy-Calibrated Redistribution then reallocates the verifier component using the teacher-to-policy token gap while preserving total task credit. On five math benchmarks, Qwen3-1.7B and Qwen3-4B students reach 17.21% and 65.09% Avg@12, beating the strongest baseline at each scale by 0.89 and 0.56 points.
- UECR-GRPO merges verifier rewards and teacher signals in one GRPO update.
- Path-Utility Unification combines both rewards before normalization and clipping.
- Entropy calibration redistributes credit while preserving the response total.
- Qwen3-1.7B hits 17.21% Avg@12, 0.89 points above the best baseline.
- Qwen3-4B hits 65.09% Avg@12, 0.56 points above the best baseline.
- AI models
- Qwen3-1.7BQwen3-4B
From the original article
1 min read at arxiv.orgReinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios.
By Jie Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.