Paper analyzes continuous-time policy gradients from discrete data
Original title: Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
The model-free actor-critic methods use PhiBE and are tested on a linear-quadratic regulator and a wheeled robot.
The paper builds a policy-gradient framework for continuous-time reinforcement learning when dynamics follow stochastic differential equations but observations arrive only at discrete times. It gives a performance difference lemma and gradient formulas for randomized and parametrized policies, then proves exponential convergence of an entropy-regularized policy flow. A bound for discrete parametrized updates separates critic error, policy-regression error and step size. PhiBE supplies a continuous-time surrogate generator so on-policy and off-policy actor-critic algorithms can learn without known dynamics.
- Experiments cover known dynamics and data-driven dynamics.
- The convergence bound isolates critic error, regression error and step size.
- Both on-policy and off-policy actor-critic variants are proposed.
From the original article
1 min read at arxiv.orgContinuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data.
By Martin Hernandez · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.