FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience
FlowBalance is a verifier-grounded self-improvement method that beats FlowRL on Qwen3-4B and Qwen3-8B math reasoning while improving training stability.
FlowBalance calibrates dense self-guidance scores with verifier-derived group advantages: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when rollout groups show no outcome preference. The method exponentially reweights a reference policy via trajectory balance, with guarantees including within-group contrast preservation and a minimum-change reverse-KL characterization. On mathematical reasoning it outperforms FlowRL on Qwen3-4B and Qwen3-8B, trains faster and more stably, avoids direct OPSD's response-length collapse, and shows higher correct-strategy diversity on AIME24.