A Unified Bellman Operator for Safety-Critical Reinforcement Learning
A unified Bellman operator learns safety-critical policies with near-zero test-time constraint violations.
The paper proposes a Bellman operator that folds task performance and hard safety constraints into one joint value function for safety-critical reinforcement learning. Temporal-difference learning with that operator is shown to converge under two-timescale stochastic approximation: safety value is estimated quickly and joint value slowly. The limiting policy theoretically maximizes task return while satisfying safety at all times, rather than only on average. Neural approximations on continuous-control tasks converge stably with near-zero safety violations at test time.
- A joint Bellman operator combines task performance and strict safety constraints.
- Two-timescale temporal-difference learning is proved to converge.
- The limiting policy maximizes return while remaining safe at all times.
- Neural continuous-control tests show near-zero safety violations at test time.
Full article176 words · extracted from arxiv.org · click to collapse
Reinforcement learning in safety-critical domains requires maximizing task performance while strictly adhering to safety constraints. Existing safe reinforcement learning paradigms typically force a trade-off: they either require a priori knowledge to provide strict safety guarantees (e.g., safety filters), or they enable joint learning but only satisfy safety constraints on average. In this work, we propose a novel Bellman operator that unifies performance and safety objectives into a joint value function. We show that temporal difference learning with the joint Bellman operator converges under a two-timescale stochastic approximation framework. On the fast timescale, the safety value of the learning joint policy is estimated, while the joint value is estimated on the slow timescale. Convergence is ensured by formulating the limiting dynamics as an occupation-averaged differential inclusion, and showing that it asymptotically converges to a set of limiting optimal safety-constrained task value functions. Theoretically, once converged, the resulting optimal policy maximizes task return while maintaining safety at all times. Empirical evaluations on continuous control tasks with neural approximations demonstrate stable convergence with near-zero safety violations at test time.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.12420