Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
DRIVE, a diversity-based RL fine-tune, lifts vision-language-action out-of-domain success, including +9.2 points on real dual-arm robots.
The paper introduces DRIVE, which turns diversity among successful behaviors into an intrinsic reward during reinforcement-learning fine-tuning of vision-language-action policies. Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, average out-of-domain performance rises by 5.3 points on pi0 and 2.0 points on pi0.5 versus vanilla RL fine-tuning. On a dual-arm AgileX PiPER-X platform, average out-of-domain success increases from 64.1% to 73.3%.
- DRIVE rewards diversity among successful trajectories, not failures.
- Average OOD gain is 5.3 points on pi0 and 2.0 on pi0.5.
- Evaluated on LIBERO-Plus, ManiSkill3, and RoboTwin 2.0.
- AgileX PiPER-X OOD success rose from 64.1% to 73.3%.
Full article174 words · extracted from arxiv.org · click to collapse
Reinforcement learning (RL) fine-tuning improves vision-language-action (VLA) policies through closed-loop experience, yet generalization beyond the fine-tuning distribution remains limited. Our analysis reveals a selective reshaping of exploration: RL contracts behavior globally, yet diversifies successful trajectories, elicits success with fewer rollouts, and covers more of the latent task-valid solution space than supervised fine-tuning. Broader successful-mode coverage may provide alternative strategies under distribution shifts. Inspired by this, we introduce DRIVE (Diversity-driven RL fIne-tuning for VLA gEneralization), which turns successful-behavior diversity into an explicit RL objective. DRIVE groups rollouts under matched task conditions, compares their trajectories with temporal alignment, and derives a success-conditioned intrinsic reward from relative behavioral diversity. This design encourages broader coverage of feasible solutions without rewarding diverse failures or superficial timing differences. Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, DRIVE improves the average out-of-domain (OOD) performance over vanilla RL fine-tuning by 5.3 points on $π_0$ and 2.0 points on $π_{0.5}$. On a dual-arm AgileX PiPER-X platform, DRIVE further increases average OOD success from 64.1% to 73.3% (+9.2 points), demonstrating gains that persist under physical deployment.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.09943