When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment
UECR-GRPO blends teacher and verifier credit, slightly improving Qwen3 math-reasoning accuracy.
The paper introduces UECR-GRPO, which combines verifier rewards and teacher guidance inside a single GRPO-style update for mathematical reasoning. Path-Utility Unification mixes a length-normalized teacher score with the verifier reward before group normalization and PPO clipping. Entropy-Calibrated Redistribution then reallocates the verifier component using the teacher-to-policy token gap while preserving total task credit. On five math benchmarks, Qwen3-1.7B and Qwen3-4B students reach 17.21% and 65.09% Avg@12, beating the strongest baseline at each scale by 0.89 and 0.56 points.
- UECR-GRPO merges verifier rewards and teacher signals in one GRPO update.
- Path-Utility Unification combines both rewards before normalization and clipping.
- Entropy calibration redistributes credit while preserving the response total.
- Qwen3-1.7B hits 17.21% Avg@12, 0.89 points above the best baseline.
- Qwen3-4B hits 65.09% Avg@12, 0.56 points above the best baseline.
Full article210 words · extracted from arxiv.org · click to collapse
Reinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios. However, teacher guidance enters after verifier-based group normalization, and token reweighting need not preserve the total task credit assigned to each response. We introduce Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), which integrates verifier and teacher signals within a single GRPO-style update at both the response and token levels. \emph{Path-Utility Unification} (PUU) combines verifier reward and a teacher-to-anchor path log-ratio in a single KL-regularized objective. Its on-policy implementation uses a length-normalized teacher score and combines both rewards before group normalization and PPO clipping, allowing teacher evidence to influence the response ranking. \emph{Entropy-Calibrated Redistribution} (ECR) then uses the signed teacher--old-policy token gap to redistribute the verifier-derived component. Full-vocabulary teacher entropy attenuates uncertain guidance, while a response-wise zero-sum projection preserves the total task credit and its token-wise sign before clipping. Across five mathematical reasoning benchmarks, UECR-GRPO achieves average \(\mathrm{Avg@12}\) accuracies of 17.21\% and 65.09\% with Qwen3-1.7B and Qwen3-4B students, respectively, exceeding the strongest baseline at each scale by 0.89 and 0.56 percentage points.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.28385