From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention
PARTS fine-tunes pretrained robot policies with RL on targeted bottleneck subtasks, boosting long-horizon manipulation success on YAM and Franka robots with minimal human intervention.
PARTS (Policy Adaptation with RL on Targeted Subtasks) is a real-world reinforcement learning framework that concentrates training on bottleneck subtasks where a frozen pretrained robot foundation policy repeatedly fails. On bimanual YAM tasks it raised complete-task success from 32% to 61%, and on single-arm Franka tasks from 50% to 95%, using tens of minutes of real-world RL rollouts per task. Agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards, enabling learning even when full-task successes are scarce. Under the same robot-rollout budget, PARTS exceeds existing real-world RL fine-tuning methods by more than 25% full-task success while requiring less human involvement.
- Subtask RL targets bottleneck steps where pretrained robot policies fail
- YAM bimanual success rose from 32% to 61%; Franka from 50% to 95%
- Requires only tens of minutes of real-world RL rollouts per task
- Beats existing real-world RL fine-tuning by over 25% under same rollout budget
Full article216 words · extracted from huggingface.co · click to collapse
A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards. We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention. The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards. These rewards support learning from successful subtasks even when complete-task successes are scarce. Training combines online RL with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. Humans identify bottlenecks during setup and perform physical resets when needed. On bimanual YAM and single-arm Franka tasks, PARTS improves complete-task success from 32% to 61% and from 50% to 95%, respectively, using tens of minutes of real-world RL rollouts per task on average. Compared with existing real-world RL fine-tuning methods, PARTS raises full-task success by more than 25% under the same robot-rollout budget while requiring less human involvement.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.21788