Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport
OptiFlow learns one-step multimodal flow policies for offline RL via state-wise entropic optimal transport, avoiding critic overestimation and mode collapse.
The paper introduces OptiFlow, a framework that frames one-step flow policy learning as a structured sample-allocation problem in offline reinforcement learning. It jointly trains a value-aware reference flow policy and a one-step policy, coupling action samples through state-wise entropic optimal transport where critic values set distillation priority and action-distance cost ensures geometrically compatible pairings. By avoiding direct critic maximization, it anchors the policy to high-value dataset-supported modes without out-of-distribution divergence. Code is released on GitHub and the method performs strongly across diverse offline RL benchmarks.
- One-step flow policy learned as structured sample allocation via entropic optimal transport
- Avoids direct critic maximization, preventing mode collapse and OOD divergence
- Critic values prioritize distillation targets; action-distance cost ensures compatible pairings
- Code publicly available; strong results across offline RL benchmarks
From the original article
1 min read at arxiv.orgOffline reinforcement learning aims to learn a policy solely from fixed datasets, which often contain multimodal action distributions. Flow policies can naturally represent such multimodal behaviors, but learning an efficient one-step flow policy remains challenging: standard value guidance often leads to mode collapse or exploits overestimation bias in out-of-distribution regions.
By Jaehun Shon · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.