ProAS reshapes advantages for multi-turn agent reinforcement learning
Original title: ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning
On Qwen2.5 instruct models, reported peaks reach 94.3% on ALFWorld and 70.8% on WebShop.
ProAS is a critic-free, plug-and-play framework that uses an Agentic Reference Graph for finer credit assignment in multi-turn LLM agents. It applies Masking Unreasonable Incentives and Asymmetric Penalty Redistribution. Experiments on Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct report peak success of 91.9% and 94.3% on ALFWorld, and 66.7% and 70.8% on WebShop. The authors say training can converge up to 2.44 times faster and remain robust to imperfect or distilled graphs.
Unconfirmed: Peak rates and the 2.44-times speedup are the authors' reported results, not independently verified here.
- ProAS targets uniform credit assignment in algorithms such as GRPO.
- Peak ALFWorld rates are 91.9% and 94.3% for the 1.5B and 7B models.
- Peak WebShop rates are 66.7% and 70.8% respectively.
- Code is released at github.com/LINs-lab/ProAS.
From the original article
1 min read at huggingface.co
Reinforcement learning (RL) has emerged as the primary paradigm for optimizing Large Language Model (LLM) agents, yet efficient critic-free algorithms (e. g. , GRPO) suffer from a fundamental uniform credit assignment dilemma that broadcasts trajectory-level advantages equally to all actions.
By Bowen Ding, Mingda Wan, Yuhan Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.