Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use
Critical-State RL identifies which multi-turn tool-use calls to train, improving missing-function task performance about 14 points on BFCL v4.
The paper introduces Critical-State RL, a method to diagnose which model calls in multi-turn tool-use interactions are trainable, using nested sampling to separate action-dependent reward variation from continuation noise. Selected states are optimized with contextual-bandit training. On Berkeley Function Calling Leaderboard (BFCL) v4, training diagnostic-selected responses improves performance by roughly 14 percentage points on missing-function tasks, while training alternative responses leaves performance flat or worse. The recipe is applied across models and tasks including logged repeat-call avoidance and memory management.
- Nested sampling separates action-dependent reward variation from downstream continuation noise.
- Contextual-bandit training at selected states beats training at alternative states.
- About 14 percentage point gain on missing-function tasks on BFCL v4.
- Recipe transfers across models, including repeat-call avoidance and memory management.
Full article191 words · extracted from arxiv.org · click to collapse
Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions. Given task-defined candidate calls and local rewards, the method assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible. It then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states using contextual-bandit training. Experiments on the Berkeley Function Calling Leaderboard (BFCL) v4 compare training at diagnostic-selected states with training at alternative states. For missing-function tasks, the diagnostic selects the response after the tool becomes available; for missing-argument tasks, it selects the response before the missing argument is supplied. Training the selected responses improves performance, including about 14 percentage points on the missing-function task, while training the alternatives leaves performance flat or worse. We further apply the recipe across models and tasks, including logged repeat-call avoidance and memory management.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.24985