Bellman Policy Optimization
Bellman Policy Optimization is a critic-free RLVR method that reformulates Policy Mirror Descent as a trajectory-level objective for LLM reasoning training.
BPO derives from Policy Mirror Descent and uses the Bellman equations to reformulate it as a trajectory-level objective for autoregressive generation with terminal rewards, avoiding state-value estimation at intermediate states. The authors prove it shares the same unique optimal solution as the original PMD objective. Experiments on mathematical reasoning benchmarks show the practical loss, built on a mismatch-correction weight over complementary token probabilities, is effective.
- Critic-free RL method derived from Policy Mirror Descent
- Uses Bellman equations to avoid estimating intermediate state values
- Proven to share PMD's unique optimal solution
- Improves LLM math reasoning under verifiable rewards
Full article103 words · extracted from huggingface.co · click to collapse
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.15987