Learning to Coach for Experiential Learning
Learning to Coach trains a dedicated LLM coach to extract transferable experiential knowledge from a frozen actor's trajectories, beating self-refinement.
Learning to Coach (L2C) trains an LLM-as-a-Coach to extract actionable experiential knowledge from a frozen actor model's previous solution trajectories, optimizing rewards based on the actor's guided response correctness. It studies same-instance and cross-instance rewards, where cross-instance elicits knowledge that transfers to other problems. Across mathematical reasoning and interactive text-games, L2C outperforms self-refinement and untrained coaches, scales better with extra inference iterations than larger decoding budgets, and transfers to out-of-distribution tasks.
- Frozen actor paired with trained LLM-as-a-Coach extracting trajectory knowledge
- Same-instance and cross-instance rewards drive guided-response correctness
- Outperforms self-refinement on math reasoning and text-games
- Extra iterations beat enlarging actor decoding budget; transfers out-of-distribution
Full article151 words · extracted from arxiv.org · click to collapse
Language models can learn from experience, but raw solution trajectories are often too long and noisy to provide effective guidance. In this work, we propose Learning to Coach (L2C), a framework that trains a dedicated LLM-as-a-Coach to extract actionable experiential knowledge from an actor model's previous trajectory. The actor remains frozen, while the LLM-as-a-Coach is trained to maximize a reward given by the correctness of the actor's guided response. We study two such rewards: a same-instance reward, which improves subsequent responses on the original problem, and a cross-instance reward, which elicits knowledge that transfers to other instances. Across mathematical reasoning and interactive text-games, L2C consistently outperforms self-refinement and an untrained LLM-as-a-Coach. Running experiential learning for more iterations further improves accuracy and uses additional inference compute more effectively than enlarging the actor's decoding budget. The trained LLM-as-a-Coach also transfers to out-of-distribution tasks and adapts its guidance to the specific actor it coaches.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.15851