Rank Collapse Is Recoverable, Growing $|Q|$ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic
SAC study finds early |Q| growth predicts value-divergence flags, while rank collapse stays recoverable.
The paper separates representation collapse from growing critic value magnitude in high update-to-data Soft Actor-Critic using width-2048 critics without normalization. At UTD ratio 16, mostly dormant HalfCheetah critics keep learning and are never flagged by a loss or |Q| explosion guard. Early log10|Q| growth by step 15,000 ranks later flags, with out-of-sample AUC 0.78 on Walker2d and 0.98 on Ant at UTD ratio 4. Aborting on that rate saves about one tenth of held-out Walker2d compute, while LayerNorm lowers the rate and removes the Walker2d flag but also lowers return.
- Dormant HalfCheetah critics at UTD 16 keep learning and stay unflagged.
- Early log10|Q| growth by step 15k ranks later guard flags.
- Out-of-sample AUC is 0.78 on Walker2d and 0.98 on Ant.
- Aborting on the growth rate saves about one tenth of Walker2d compute.
- LayerNorm lowers the growth rate and Walker2d return at UTD 4.
From the original article
1 min read at arxiv.orgRaising the update-to-data (UTD) ratio breaks off-policy critics in two ways grouped as "plasticity loss": collapsing representations and growing value magnitude $|Q|$. We separate them in Soft Actor-Critic (SAC) with scaled critics (width 2048, no normalization).
By Tianqi Bu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.