Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
DARA reweights multi-reward RL by reward density, reaching compliance faster than GDPO on tools and math.
Multi-reward reinforcement learning can leave language-model objectives with uneven progress even after reward-wise normalization such as GDPO. The authors show that, under idealized GDPO normalization, a reward's advantage energy is proportional to active-group density, the fraction of rollout groups with nonzero relative advantages. Density-Aware Reward Aggregation applies a per-batch inverse-square-root density correction that gives more weight to less frequently active rewards without changing the policy objective. On tool calling and mathematical reasoning, DARA reaches high format compliance in up to 26% fewer steps and near-saturated length compliance in up to 65% fewer steps than GDPO, with competitive final performance.
- Under idealized GDPO, advantage energy scales with active-group density.
- DARA upweights sparse rewards using an inverse-square-root density correction.
- Tool-calling format compliance arrives in up to 26 percent fewer steps.
- Math length compliance nears saturation in up to 65 percent fewer steps.
From the original article
1 min read at huggingface.co
Multi-reward reinforcement learning trains large language models to satisfy multiple behavioral objectives simultaneously. Reward-wise normalization, as used in GDPO, preserves reward-specific relative information within rollout groups, but different objectives can still exhibit uneven learning progress. We study this behavior through advantage energy, the sum of a reward's squared advantages over a batch.
By Tong Zheng, Skylar Zhai, Zhan Cheng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.