Hugging Face daily papers·11d agoRethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening#critic-learning#ppo#preference-optimization