Near-Optimal Reinforcement Learning with Multi-Step Transition Lookahead
Theorists prove multi-step lookahead RL planning is NP-hard for every fixed rational discount factor yet give a randomized polynomial-time approximation scheme.
The paper resolves open questions about reinforcement learning with multi-step transition lookahead. It shows exact planning remains NP-hard for every fixed rational discount factor in (0,1), not just discounts arbitrarily close to one, and introduces a randomized polynomial-time approximation scheme for every fixed lookahead depth. Extending to unknown transitions and stochastic rewards via optimism and variance-adaptive confidence bounds, the algorithm achieves cumulative regret matching classical tabular discounted RL up to logarithmic factors.
- NP-hardness of exact lookahead planning now holds for every fixed rational discount factor.
- Randomized PTAS achieves near-optimal planning for any fixed lookahead depth.
- Optimism and variance-adaptive bounds extend results to unknown transitions and stochastic rewards.
- Regret matches classical tabular discounted RL up to logarithmic factors.
From the original article
1 min read at arxiv.orgWe study reinforcement learning (RL) with transition look-ahead, where the agent may observe which states would be visited upon playing any sequence of $\ell$ actions before deciding its course of action.
By Corentin Pla · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.