Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL
GAGAR uses groupwise agentic grading to shift code-agent RL credit toward cleaner implementations.
GAGAR is a quality-aware credit scheme for code-agent reinforcement learning. After dynamic sampling keeps groups with both passing and failing trajectories, an SFT-trained agentic grader ranks test-passing solutions and downweights lower-ranked ones, then rescales advantages so their original sum is preserved. Controlled code-only experiments on a pre-RL MiMo-V2.6-Flash checkpoint (310B total parameters) improved code-agent performance, reduced trajectory-length growth, and stabilized training. The method was also applied in large-scale mixed-task RL with Flash and MiMo-V2.6-Pro (1.02T).
- Binary test rewards give identical GRPO advantages to every passing trajectory.
- An SFT grader ranks passing candidates and redistributes advantages while preserving their sum.
- Evaluated on MiMo-V2.6-Flash (310B) and MiMo-V2.6-Pro (1.02T).
- Code-only Flash runs improved agent quality, shortened trajectories, and stabilized training.
- AI models
- MiMo-V2.6-FlashMiMo-V2.6-Pro
From the original article
1 min read at huggingface.co
Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and adherence to task requirements.
By Jinhao Dong, Liang Zhao, Zihao Yue · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.