Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL
MRTPO assigns each reward's advantage only to tokens relevant to that verifier, improving multi-reward RL.
Language models trained with several verifiable rewards still spread each reward's advantage across the whole response under reward-decoupled normalization (GDPO). MRTPO measures per-token Jensen-Shannon distance by replaying responses with and without each verifier's information, then reassigns advantages without auxiliary models or token annotations. On text-to-SQL, multi-constraint instruction following, and tool calling, it converges faster and outperforms compared methods.
- Assigns each reward advantage only to verifier-relevant tokens
- Uses Jensen-Shannon distance between next-token distributions
- Needs no auxiliary models or token-level annotations
- Improves text-to-SQL, instruction following, and tool calling
From the original article
1 min read at huggingface.co
Language models are increasingly trained against several verifiable rewards at once, such as the correctness and the format of a tool call, or the individual constraints of a complex instruction.
By Skylar Zhai, Haonan Wang, Zhining Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.