Training Advisors for LLM Agents from Task Outcomes
Caddie trains an outcome-based critic that lifts LLM agent success, including a 25-point gain for Qwen3-4B.
Caddie trains a critic to give natural-language advice to LLM agents, using reinforcement learning from whether the agent ultimately succeeds and keeping the base model frozen. A Qwen3-4B critic trained on multi-hop question answering improves four base models of different scales and architectures, including three not seen in training. On MuSiQue it raises Qwen3-4B success by more than 25 percentage points, above Kimi K3 without a critic, and also improves tau-3 and DeepDive with no further training.
- Critic is optimized only from final task success with the base model frozen.
- A Qwen3-4B critic transfers to three models unused in training.
- MuSiQue success rises more than 25 points, above critic-free Kimi K3.
- The same critic helps on tau-3 and DeepDive without extra training.
Full article200 words · extracted from arxiv.org · click to collapse
Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including $τ^3$ and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.09858