MInTRL: Off-policy Intervention can boost On-policy RL
MInTRL injects sparse judge corrections into on-policy RL rollouts, expanding exploration beyond on-policy sampling while preserving learnability on math and code benchmarks.
Minimal Intervention Reinforcement Learning periodically has a judge-intervention policy replace erroneous suffixes of the current policy's output with short corrections, then returns control, keeping trajectories largely on-policy. Training uses a sequence-level advantage-regression objective that removes the need for importance sampling. Across math and code benchmarks it consistently beats standard on-policy and off-policy baselines, remains effective with self-intervention, and performs best at moderate intervention intensity.
- Sparse, local interventions expand exploration frontier of on-policy RL with verifiable rewards.
- Judge replaces erroneous suffixes mid-generation and immediately returns control to the policy.
- Sequence-level advantage-regression objective eliminates importance sampling.
- Outperforms on-policy and off-policy baselines; peaks at moderate intervention intensity.
Full article203 words · extracted from huggingface.co · click to collapse
Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts. During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines. Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.12419