FewStepRL adapts diffusion RL objectives to few-step samplers
Original title: FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning
The method recasts FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL so rollouts no longer need a many-step sampler.
FewStepRL adapts reinforcement-learning objectives for few-step diffusion models, which predict average velocity over an interval rather than instantaneous velocity. The authors use a transformation to a shared policy prediction, a gradient decomposition into self-regression and reward augmentation, and anchors computed from the few-step model. Reward-gradient methods were strongest when the reward was differentiable and its gradient affordable; otherwise DiffusionNFT-style methods generally reached high rewards in relatively few iterations.
Unconfirmed: Effectiveness rankings are from the authors' experiments only.
- Objectives are expressed through one policy prediction, clean-end by default.
- Each gradient splits into a reference self-regression term and a reward term.
- Instantaneous velocity comes from the network itself or an auxiliary network.
- Extra inference steps showed no systematic aggregate benefit in their experiments.
From the original article
1 min read at huggingface.co
Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training.
By Haitao Lin, Minsi Ren, Yang Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.