Routing Policy Optimization for Online Diffusion Reinforcement Learning
RPO uses attention-route exploration to align diffusion generators about 4x more efficiently than Flow-GRPO.
Routing Policy Optimization (RPO) aligns diffusion generators with reward signals by sampling attention routes at selected denoising levels instead of perturbing the reverse trajectory. A controlled two-rollout estimator scores route perturbations with a score-function objective, and learned updates remain effective after routing is removed. At inference, dense attention and the original deterministic sampler are restored. On text-to-image and text-to-video tasks, RPO improves quality with nearly 4x higher training efficiency than Flow-GRPO-style methods.
- RPO treats attention routing as stochastic exploration inside the generator.
- A two-rollout estimator uses reward differences between sampled routes.
- Dense attention and the original deterministic sampler return at inference.
- Reported training efficiency is nearly 4x Flow-GRPO-style methods.
From the original article
1 min read at huggingface.co
Reinforcement learning can align visual generators with reward signals beyond their pretraining objectives, but existing score-function methods typically introduce stochasticity by perturbing the reverse denoising trajectory, leading to costly rollout-based optimization. We instead ask whether stochastic exploration can be introduced inside the generator while preserving its original sampler.
By Qiuhong Shen, Yiding Yang, Yuxin Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.