A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
Paper unifies regularization-based robust RL methods via new performance-gap upper bounds and jointly learned Lagrange multipliers.
The authors derive new upper bounds on the gap between nominal and worst-case deep RL policies, each expressible as an existing regularization objective plus a KL-divergence penalty. Robust training is reformulated as constrained optimization, where prior methods correspond to a fixed Lagrange multiplier. The multiplier is instead updated jointly with the policy, auto-tuning the regularization weight. Adversarial evaluations across several continuous control tasks validate the theory.
- New upper bounds link regularization objectives to worst-case performance gaps
- KL penalty between nominal and worst-case policies explains robustness gains
- Lagrange multiplier learned jointly with policy for automatic weight tuning
- Validated with adversarial evaluations on continuous control tasks
Full article129 words · extracted from arxiv.org · click to collapse
Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.13050