FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience
FlowBalance is a verifier-grounded self-improvement method that beats FlowRL on Qwen3-4B and Qwen3-8B math reasoning while improving training stability.
FlowBalance calibrates dense self-guidance scores with verifier-derived group advantages: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when rollout groups show no outcome preference. The method exponentially reweights a reference policy via trajectory balance, with guarantees including within-group contrast preservation and a minimum-change reverse-KL characterization. On mathematical reasoning it outperforms FlowRL on Qwen3-4B and Qwen3-8B, trains faster and more stably, avoids direct OPSD's response-length collapse, and shows higher correct-strategy diversity on AIME24.
- Calibrates self-guidance with verifier group advantages on on-policy rollouts
- Reverses guidance on negative-advantage trajectories; disables with no outcome preference
- Beats FlowRL on Qwen3-4B and Qwen3-8B math reasoning
- Avoids OPSD response-length collapse; higher strategy diversity on AIME24
- Tags
- math-reasoningqwen3reasoning-modelsreinforcement-learningself-improvementtrajectory-balanceverifiers
- Products
- FlowBalanceFlowRLOPSD
From the original article
1 min read at huggingface.co
A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses.
By Zixun Huang, Kishan Panaganti, Haitao Mi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.