Bellman Policy Optimization
Bellman Policy Optimization is a critic-free RLVR method that reformulates Policy Mirror Descent as a trajectory-level objective for LLM reasoning training.
BPO derives from Policy Mirror Descent and uses the Bellman equations to reformulate it as a trajectory-level objective for autoregressive generation with terminal rewards, avoiding state-value estimation at intermediate states. The authors prove it shares the same unique optimal solution as the original PMD objective. Experiments on mathematical reasoning benchmarks show the practical loss, built on a mismatch-correction weight over complementary token probabilities, is effective.
- Critic-free RL method derived from Policy Mirror Descent
- Uses Bellman equations to avoid estimating intermediate state values
- Proven to share PMD's unique optimal solution
- Improves LLM math reasoning under verifiable rewards
From the original article
1 min read at huggingface.co
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective.
By Zhuoqing Song, Haotian Xu, Xikun Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.