iADD: Improving Alignment and Diversity in Diffusion Policy Optimization
iADD improves the alignment-diversity tradeoff in diffusion-model post-training using incremental Feynman-Kac updates.
The paper studies reinforcement-learning post-training of diffusion models, including Denoising Diffusion Policy Optimization, where reward optimization often reduces diversity and quality. The authors show mathematically that updating only later timesteps can harm diversity, contrary to earlier conclusions. They propose incremental Feynman-Kac training and compare it with related diffusion policy optimization methods on three tasks, with ablations supporting improvements in both alignment and diversity.
- Later-timestep-only diffusion updates can harm diversity, contrary to prior work.
- Incremental Feynman-Kac training targets a better alignment-diversity tradeoff.
- Comparisons on three tasks show gains in both alignment and diversity.
From the original article
1 min read at huggingface.co
Reinforcement learning based post training of diffusion models, such as Denoising Diffusion Policy Optimization (DDPO), optimizes a reverse diffusion process under a reward function. However, current approaches to reward optimizations do so at the cost of diversity and quality. In this paper, we provide better tradeoffs through careful theoretical considerations and method design.
By Ashok Prasad Neupane, Saugat Adhikari, Pramish Paudel · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.