CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
CorrGRPO normalizes multi-reward GRPO with Pearson correlations so large rewards do not drown smaller signals.
CorrGRPO changes multi-reward GRPO so pairwise reward covariances are normalized as Pearson correlations instead of raw covariance scale. The centered total reward stays unchanged, so advantage size can reflect reward dependence without large-scale rewards drowning smaller signals. On models from 0.5B to 8B it improves code generation, tool calling, and agent-security tasks versus GRPO and other variants. Code is released by HKUST-KnowComp.
- Normalizes pairwise covariances as Pearson correlations
- Stops large-scale rewards from dominating advantage magnitudes
- Gains on code generation, tool calling, and agent security
- Evaluated on models from 0.5B to 8B parameters
- Organizations
- HKUST-KnowComp
From the original article
1 min read at huggingface.co
Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances.
By Wenbin Hu, Huihao Jing, Haochen Shi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.