$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource
Lambda-Controlled GRPO budgets flow-matching path variance so image-model RL stays stable and improves rewards.
Flow-GRPO optimizes flow-matching image generators from reward feedback, but importance ratios drift across denoising steps and late steps leave fewer usable samples. The authors attribute these effects to path variance, which is fixed by the sampler's Gaussian transition kernel and can be estimated during training. Lambda-Controlled GRPO calibrates importance ratios from that predicted law and allocates gradient effort by predicted cost, without new free tuning parameters. On text-to-image tasks scored by OCR and a preference model, it improves both rewards and keeps late-step variance within budget.
- Flow-GRPO instability is traced to one per-step path-variance quantity.
- Update scales follow policy choices rather than extra free hyperparameters.
- OCR text accuracy and preference reward beat the strongest empirical stabilizer.
- Late-step path variance stays inside budget where the baseline overshoots.
Full article271 words · extracted from arxiv.org · click to collapse
Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback. Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer. We show instead that they arise from a single per-step quantity, which we call path variance. This quantity is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a resource that can be measured and budgeted rather than a collection of symptoms to repair. Our method, $λ$-Controlled GRPO, calibrates importance-ratio behavior from this predicted law rather than from noisy empirical statistics, and allocates gradient effort across denoising steps according to their predicted cost. The two scales governing the update are fixed by standard policy choices rather than introduced as free tuning parameters. On a text-to-image model under two reward settings, rendering difficult target text scored by optical character recognition and matching human preferences scored by a preference model, $λ$-Controlled GRPO improves both text accuracy and preference reward over the strongest empirical stabilizer. It also keeps late-step path variance within its intended budget, precisely where the baseline systematically overshoots. The result is a Flow-GRPO update calibrated by its own transition law rather than stabilized after instability appears.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.22041