MInTRL: Off-policy Intervention can boost On-policy RL
MInTRL injects sparse judge corrections into on-policy RL rollouts, expanding exploration beyond on-policy sampling while preserving learnability on math and code benchmarks.
Minimal Intervention Reinforcement Learning periodically has a judge-intervention policy replace erroneous suffixes of the current policy's output with short corrections, then returns control, keeping trajectories largely on-policy. Training uses a sequence-level advantage-regression objective that removes the need for importance sampling. Across math and code benchmarks it consistently beats standard on-policy and off-policy baselines, remains effective with self-intervention, and performs best at moderate intervention intensity.
- Sparse, local interventions expand exploration frontier of on-policy RL with verifiable rewards.
- Judge replaces erroneous suffixes mid-generation and immediately returns control to the policy.
- Sequence-level advantage-regression objective eliminates importance sampling.
- Outperforms on-policy and off-policy baselines; peaks at moderate intervention intensity.
From the original article
1 min read at huggingface.co
Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift.
By Mingyu Chen, Yefan Tao, Gerald Friedland · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.