ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning
ReSPO replaces clipping in off-policy RL to fix gradient starvation on reused rollouts for Qwen3.
The paper identifies sign-dependent gradient starvation in clipped policy optimization when reinforcement learning from verifiable rewards reuses rollouts across updates. ReSPO replaces clipping with a smooth two-branch sequence-level kernel derived from an alpha-divergence objective and an exponential variance-control tilt. The positive branch preserves gradient on under-generated correct responses, while the negative branch down-weights severely over-generated negatives. On dense and mixture-of-experts Qwen3 models, ReSPO speeds early training and raises final and held-out scores under rollout reuse.
- Clipping starves under-generated positive responses when rollouts are reused
- ReSPO replaces clipping with a two-branch sequence-level kernel
- Positive branch keeps gradient on long correct reasoning trajectories
- Negative branch suppresses heavily over-generated wrong responses
- Improves dense and MoE Qwen3 scores and held-out benchmarks
From the original article
1 min read at huggingface.co
Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy.
By Yihang Chen, Yuanhao Ban, Cho-Jui Hsieh · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.