Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
RGPO uses temporary rationale scaffolds to cut reward sparsity in reasoning RL for text and multimodal models.
Rationale-Guided Policy Optimization (RGPO) addresses sparse rewards in on-policy reinforcement learning for LLM reasoning. Ground-truth rationales temporarily scaffold generation, then only higher-reward model solutions are transferred back to the original unguided task, avoiding format-matched off-policy data. The paper reports consistent gains over RLVR baselines in text-only and vision-language settings, with ablations attributing gains to adaptive rationale guidance.
- RGPO treats ground-truth rationales as temporary scaffolds, not fixed imitation targets.
- Only higher-reward model-generated solutions return to the unguided RL setting.
- It beats RLVR baselines on language-only and vision-language reasoning.
- Ablations identify adaptive rationale guidance as a main source of gains.
From the original article
1 min read at huggingface.co
On-policy reinforcement learning has become a central paradigm for improving the reasoning abilities of large language models. However, its effectiveness is often limited by reward sparsity: when a model fails to discover correct trajectories for difficult problems, the optimization process receives little useful signal and may stagnate.
By Hoang Phan, Minh Pham, Chau Pham · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.