Parametric resonance can destabilize primal-dual learning in constrained MDPs
Original title: Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs
Mismatched constraint estimates drive resonance at twice the natural frequency; a shared estimate removes it but adds bias.
The paper analyzes primal-dual methods for constrained Markov decision processes when constraints change slowly. Additive oscillation of the constraint level saturates dual oscillation at a floor and amplifies it with the timescale ratio. Parametric oscillation of constraint sensitivity produces resonance at twice the natural frequency when primal and dual updates use mismatched estimates. In a Lagrangian soft actor-critic agent, the multiplier locks onto the predicted sub-harmonic.
- Tolerated mismatch grows with the square root of forcing bandwidth.
- It collapses with the square root of the dual-to-primal step-size ratio.
- Driving both updates from one estimate removes resonance but adds estimator bias.
- A coherent cost critic suppresses lock-in; a PID term was predicted not to.
From the original article
1 min read at arxiv.orgPrimal-dual methods for constrained Markov decision processes are analyzed under a two-timescale assumption, with non-stationarity entering through two channels. In the additive channel, where the constraint level oscillates, the dual oscillation saturates at a floor set by the moving optimum and is amplified linearly by the timescale ratio.
By Maxime Baelde · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.