Safe Meta-Reinforcement Learning via Information Space Reachability
Safe meta-RL framework reasons about safety in information space, learning a safety value function used for safety filtering and constrained policy optimization.
The paper proposes safe meta-RL that reasons about safety in information space, capturing both physical state and the agent's belief over the underlying task. A safety value function measures the probability of avoiding unsafe regions indefinitely and satisfies a self-consistency condition and Bellman equation, making it learnable via meta-RL. The resulting algorithm uses the learned function for safety filtering and constrained policy optimization, with effectiveness demonstrated on meta-RL benchmarks.
- Safety handled in information space combining physical state and task belief
- Safety value function satisfies Bellman equation, learnable via meta-RL
- Applied to safety filtering and constrained policy optimization
- Validated on standard meta-RL benchmarks
Full article153 words · extracted from arxiv.org · click to collapse
Meta-reinforcement learning (meta-RL) enables agents to adapt to unseen tasks with limited experience. Despite its promise, the application of meta-RL in real-world tasks is hindered by safety requirements, which have been underexplored in prior work. In this paper, we propose a safe meta-RL framework that explicitly accounts for safety during adaptation. Our key insight is to reason about safety in the information space, which captures both the physical state and the agent's belief over the underlying task. Within this space, we introduce a safety value function that measures the probability of the agent avoiding unsafe regions indefinitely. We show that this function satisfies a self-consistency condition and a Bellman equation, which make it learnable via meta-RL. Based on this formulation, we develop a safe meta-RL algorithm that learns the safety value function and leverages it for safety filtering and constrained policy optimization. Experiments on meta-RL benchmarks demonstrate the effectiveness of the proposed method.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.15915