Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping
Retimed Bellman Flows cut return-distribution error up to 7.7× and beat prior flow critics on 38 offline RL tasks.
Retimed Bellman Flows (ReBF) address a bias problem in velocity bootstrapping for flow critics that learn return distributions. By querying the teacher at a dynamically earlier flow time and using fresh decoupled noise, ReBF constructs a conditionally unbiased velocity target that preserves the Bellman fixed point. On synthetic MRPs it reduces Wasserstein-1 distance to ground-truth returns by up to 7.7× and outperforms existing flow critics on 38 OGBench and D4RL offline RL tasks.
- ReBF queries a teacher at a shifted earlier flow time.
- It builds a conditionally unbiased velocity target that preserves the Bellman fixed point.
- W1 error falls by up to 7.7× on synthetic MRPs.
- It beats prior flow critics on 38 OGBench and D4RL tasks.
Full article138 words · extracted from arxiv.org · click to collapse
Flow critics learn return distributions by transporting Gaussian noise to Bellman endpoints via continuous velocity fields. While velocity bootstrapping stabilizes training by querying a successor teacher, existing methods face a structural dilemma: on straight paths, no residual-free same-time affine mapping can preserve Gaussian initial noise while maintaining an unbiased target. To overcome this limitation, we introduce Retimed Bellman Flows (ReBF). ReBF queries the teacher critic at a dynamically shifted earlier flow time, aligning intermediate student and teacher trajectories. By combining this retimed clock with fresh, decoupled noise generation, ReBF constructs a provably conditionally unbiased velocity target that preserves the Bellman fixed point and contracts under Wasserstein distances. Empirically, ReBF reduces $W_1$ distance to ground-truth return distributions by up to $7.7\times$ on synthetic MRPs and outperforms existing flow critics across 38 challenging OGBench and D4RL offline reinforcement learning tasks.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.32828