The Low-Rank Structure of VLA Reinforcement Learning
RL post-training of flow-based VLA models produces low-rank updates concentrated in action-expert Timestep Modules.
Researchers find reinforcement learning on flow-based vision-language-action models, including pi0.5 and GR00T N1.5/N1.6, produces substantially lower-rank updates concentrated in the action expert's Timestep Modules. Tests span LIBERO, ManiSkill, MetaWorld, and CALVIN, and module-replacement experiments show these modules account for a disproportionate share of RL gains. RL specializes the modules to discrete denoising timesteps; shift-vector update directions predict task success with ROC-AUC up to 99.6% and reflect cross-task transfer. Steering along those directions further improves policies without additional RL.
- Low-rank RL updates concentrate in VLA action-expert Timestep Modules.
- Evaluated on pi0.5 and GR00T N1.5/N1.6 across four robotics suites.
- Shift-vector directions predict task success with ROC-AUC up to 99.6%.
- Steering along those directions improves policies without further RL.
Full article219 words · extracted from huggingface.co · click to collapse
Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including π_{0.5} and GR00T~N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to 99.6%). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.34599