Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Researchers identify Value Flattening in PPO critics for LLM RL and propose SP^3O sparse value supervision, improving Qwen3-Base training.
1 stories, newest first · one card per story
Researchers identify Value Flattening in PPO critics for LLM RL and propose SP^3O sparse value supervision, improving Qwen3-Base training.
Today 1.5K readers · 2.4K views · 757 reads · all time 189K views