SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL
SLCA-GRPO assigns separate advantages to tool and summary tokens, improving tool-calling reinforcement learning.
SLCA-GRPO targets credit misattribution in tool-calling reinforcement learning, where GRPO applies one trajectory-level advantage to both tool calls and natural-language summaries. Segment-Locked Credit Assignment, with hierarchical rewards, sends execution advantages to tool tokens and preference advantages to summary tokens without extra intermediate rollouts. A Schema-Guided LLM Simulator provides scalable training without live APIs. On a 7B backbone, it outperforms GRPO, ToolPO, and RLTR by 2.53, 1.36, and 9.15 percentage points on in-domain, BFCL, and τ²-Bench evaluations.
- Segment-locked advantages stop summary noise from contaminating tool tokens.
- A schema-guided simulator avoids costly live API calls during training.
- Gains are 2.53, 1.36, and 9.15 points on three evaluations.
- The 7B policy reduces redundant tool calls under the same budget.
From the original article
1 min read at huggingface.co
Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries. This output heterogeneity presents a structural failure mode in standard on-policy Reinforcement Learning (RL): algorithms like GRPO indiscriminately broadcast a homogeneous trajectory-level scalar advantage to all tokens.
By Yan Zhan, Shaobo Liu, Qiunan Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.