Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Researchers identify Value Flattening in PPO critics for LLM RL and propose SP^3O sparse value supervision, improving Qwen3-Base training.
The paper uncovers Value Flattening, a failure mode where PPO critic predictions stay flat while true state values estimated from Monte Carlo continuations change sharply, worsening as state spaces grow. The authors attribute it to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states. They propose SP^3O, which supervises value loss on only a few well-separated states per response, consistently improving policies trained on Qwen3-Base across model sizes and evaluation suites.
- Value Flattening reproduced in controlled FrozenLake environments.
- Critic loss implicitly penalizes variance and correlates redundant updates.
- SP^3O supervises only three states per response.
- Consistent policy improvements across Qwen3-Base model sizes.
- Products
- SP^3O
- AI models
- Qwen3-Base
From the original article
1 min read at huggingface.co
In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates.
By Yizhuo Li, Jianhao Yan, Yun Luo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.