PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents
PivotOPD trains multi-turn agents to avoid pivotal mistakes and recover within a few turns.
Across three Qwen3 models from 8B to 235B, more than half of failed multi-turn rollouts contain an early pivotal mistake that is often recoverable within a few guided turns. PivotOPD, from NVIDIA, uses reverse-KL distillation on a teacher gold action to prevent that mistake and forward-KL distillation on subsequent recovery actions. It records the strongest average among 13 baselines for Qwen3-1.7B and Qwen3-8B on ALFWorld, WebShop, and search-based QA, including a 5.5-point ALFWorld gain for the 1.7B student. A Nemotron-3.5 student also improves SWE-Bench Verified resolve rate by 3.2 points.
- More than half of failed Qwen3 rollouts have early pivotal mistakes
- Teacher supplies a gold action and a short recovery sequence
- Best average among 13 baselines for 1.7B and 8B students
- ALFWorld +5.5% for Qwen3-1.7B; SWE-Bench +3.2% for Nemotron-3.5
Full article251 words · extracted from huggingface.co · click to collapse
On-policy distillation (OPD) is a promising approach for training language agents, providing dense teacher supervision on student-generated trajectories. However, in multi-turn interaction, an incorrect action changes the states the student encounters later, so errors compound across turns. In preliminary experiments across three Qwen3 models (8B to 235B), we find that more than half of the failed rollouts contain a pivotal mistake, an action that moves the agent farther from completing the task, and this mistake typically occurs early. These pivotal mistakes often remain recoverable: guiding the model for only a few turns after the pivotal turn can restore task success. We therefore propose PivotOPD, an on-policy distillation framework that jointly trains the student to prevent pivotal mistakes and to recover from the states they create. At each pivotal mistake, a teacher model provides a gold action and then names a recovery action at each of the next few turns. Preventive distillation uses the gold action with reverse KL to steer the student away from the pivotal mistake, while recovery distillation uses the recovery actions with forward KL to transfer recovery behaviors that the student rarely samples. Against 13 baselines on ALFWorld, WebShop, and Search-based QA, PivotOPD achieves the strongest average performance for both Qwen3-1.7B and Qwen3-8B students, improving over the strongest baseline on ALFWorld by +5.5% with the 1.7B student. The gains also transfer to another model family on the software engineering domain, where PivotOPD raises the resolve rate of a Nemotron-3.5 student on SWE-Bench Verified by +3.2%. Project page: https://research.nvidia.com/labs/lpr/pivotopd/
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.40285