Matched compute reverses dense-reward gains over GRPO
Original title: Are Dense Rewards Worth Their Tokens in GRPO?
Across 1,668 runs, a Monte-Carlo process reward trailed GRPO by 14.3 points at equal tokens on Qwen2.5-1.5B.
A paper argues that dense process rewards for group-relative policy optimization look better per update but lose once compute is matched. Continuations for a Monte-Carlo step reward consumed 91% of generated tokens in the authors' setting. On Qwen2.5-1.5B that reward finished 14.3 points below GRPO at equal tokens and 7.8 points lower at equal training time. Selecting prompts whose predicted success is nearest one half improved accuracy at matched compute.
- None of four dense-reward variants beat GRPO at equal training time.
- Prompt selection gained 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B.
- Doubling GRPO's initial learning rate came within half a point.
- A per-token gradient test overestimated rewards by one to two points.
- Gains replicated on fresh seeds and unseen problems.
From the original article
1 min read at huggingface.co
RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit, so dense process rewards have been proposed to credit intermediate steps, for example with a Monte-Carlo reward that scores each step by sampling continuations from it.
By Vikash Singh, Weicong Chen, Debargha Ganguly · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.