Bellman theory values multi-step transition predictions in finite MDPs
Original title: Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory
BF-BOLA learns near-optimal policies with sample bounds that match up to logarithmic factors.
The paper studies discounted finite MDPs in which a decision maker receives K-step probabilistic transition predictions for some actions at the start of each block and adapts to states inside the block. Optimal value follows a Bayesian Bellman equation on the original state space, avoiding exponential growth from adding predictions to the state. Predictive information changes the order of maximization and expectation. The authors propose BF-BOLA and matching upper and lower sample-complexity bounds.
- The analysis covers horizon, error, action coverage, and within-block feedback.
- Predictions can limit transition learning to uncovered rows.
- A near-optimal policy can need fewer samples than the Bayesian value function.
From the original article
1 min read at arxiv.orgMulti-step predictions of future transition realizations can improve reinforcement learning, but their value remains difficult to quantify. Directly including these predictions in the state can lead to exponential state-space growth, making direct dynamic programming computationally expensive.
By Chenbei Lu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.