Theory says REINFORCE can collapse coverage in LM post-training
Resetting the KL reference in Approximate Policy Mirror Descent is shown to restore global convergence for stylized PPO and GRPO.
Stories per day · 30 days
2 RSS
Relatedreinforcement-learning5qwen32reasoning2self-distillation2vaspo2alfworld1computer-use-agents1game-agents1grpo1minedojo1olmo1olmo31player-11policy-optimization1post-training1scienceworld1world-models1
18 stories, newest first · one card per story
Stories tagged #ppo
WM-PPO values world-model predictions to improve sparse-reward agents
With a similar number of real steps, success rises from 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld.
Player-1: Learning to Play from Synthetic Experience
Player-1 uses synthetic RL environments to substantially improve game-control agents beyond supervised training.
VASPO: Self-Distillation with Privileged Value Estimation
VASPO combines privileged self-distillation with PPO, gaining up to 2.59 points over PPO on math benchmarks.
VASPO: Self-Distillation with Privileged Value Estimation
VASPO blends a privileged self-teacher with PPO, improving math accuracy on Qwen3-4B and OLMo3-7B.
Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning
Reinforcement learning manages deep-space bundle relays to cut excess age and improve flow fairness.
Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning
Explanatory World Models reassign delayed RL credit and beat PPO and LaRe on long-horizon tasks.
ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
ViSkill gives VLM agents reusable visual skill cards, reaching 0.89 success and 0.91 with cold start.
Energy-Aware Path Following: Comparative Analysis of Reinforcement Learning and NMPC for Electric Vehicles
Researchers compare PPO and NMPC for energy-aware EV path following, with PPO transferring to unseen tracks.
PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading
PPO-HRAP blends a regime prior with PPO, cutting SPY max drawdown to 18.47% versus 34.10% buy-and-hold.
DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation
DexPolicy anneals exploration for dexterous manipulation, raising target success for PPO, GRPO, and FPO.
CrossBFM: Distilling a Shared Latent Behavior Space Across Humanoid Embodiments
CrossBFM distills one latent behavior space that transfers across humanoid robots in under an hour of GPU training.
EasyPPO: Stabilizing the Critic Is Key
EasyPPO stabilizes the PPO critic for LLM training and outperforms vanilla PPO on coding, math, and search.
Nereus: Adaptive Parallelism for LLM Post-Training
Nereus adapts parallelism during LLM reinforcement-learning post-training, cutting step latency 27.7% versus a fixed layout.
Playing to Par: Reinforcement Learning for Provably Optimal Quadrilateral Block Decompositions
An RL agent produces provably optimal quadrilateral meshes that outperform Gmsh on held-out domains.
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Researchers identify Value Flattening in PPO critics for LLM RL and propose SP^3O sparse value supervision, improving Qwen3-Base training.
Learning Intrusion Response Strategies for OT Systems
Researchers model OT intrusion response as a POMDP and train PPO-based automated response strategies effective against MITRE attacks in an emulated OT system.
5 useful things you'll learn in my new post-training textbook (shipping now!)
Nathan Lambert's new RLHF and post-training LLM textbook covers PPO, GRPO, GSPO, CISPO and related techniques, freely available online.