Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use
Critical-State RL identifies which multi-turn tool-use calls to train, improving missing-function task performance about 14 points on BFCL v4.
The paper introduces Critical-State RL, a method to diagnose which model calls in multi-turn tool-use interactions are trainable, using nested sampling to separate action-dependent reward variation from continuation noise. Selected states are optimized with contextual-bandit training. On Berkeley Function Calling Leaderboard (BFCL) v4, training diagnostic-selected responses improves performance by roughly 14 percentage points on missing-function tasks, while training alternative responses leaves performance flat or worse. The recipe is applied across models and tasks including logged repeat-call avoidance and memory management.
- Nested sampling separates action-dependent reward variation from downstream continuation noise.
- Contextual-bandit training at selected states beats training at alternative states.
- About 14 percentage point gain on missing-function tasks on BFCL v4.
- Recipe transfers across models, including repeat-call avoidance and memory management.
From the original article
1 min read at arxiv.orgMulti-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions.
By Zixiang Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.