QF3: Fast Flow RL with Filtered Q-Gradients
QF3 trains robot flow policies off-policy, learning humanoid locomotion from scratch about ten times faster than FPO++.
Researchers introduce QF3, an online off-policy reinforcement learning algorithm that trains flow policies with flow matching plus a critic action gradient through a one-step flow prediction. Critic gradients are applied only to action dimensions that stay near the replay action, limiting updates to regions where the critic is reliable. The method is reported as the first off-policy flow RL approach to train humanoid locomotion from scratch and transfer it zero-shot to hardware, with about a 10x wall-clock speedup versus on-policy FPO++. It also fine-tunes pretrained flow manipulation policies on ABC-Sim and Robomimic.
- Off-policy flow RL applies filtered critic gradients through one-step predictions.
- Trains humanoid locomotion from scratch and transfers it zero-shot to hardware.
- Reports about a 10x wall-clock speedup over on-policy FPO++.
- Fine-tunes pretrained manipulation policies on ABC-Sim and Robomimic.
From the original article
1 min read at arxiv.orgFlow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction.
By Chung Min Kim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.