RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

RetireOPD trains multi-turn agents via self on-policy distillation with adaptive teacher retirement, improving Qwen2.5 agent success on ALFWorld by 14.1-18.8% and WebShop accuracy by 11.8-19.0% over RL baselines.

RetireOPD is a method for agentic reinforcement learning that first trains a decoupled, skill-conditioned teacher with environment rewards, then trains a skill-free student jointly with reinforcement learning and on-policy distillation, using dense token-level supervision from the self-teacher. An Adaptive Retirement mechanism drops the teacher once the student-teacher discrepancy stops shrinking and the student reaches a target fraction of the teacher's success rate. Across Qwen2.5 models from 1.5B to 7B on the ALFWorld and WebShop tasks, RetireOPD improves ALFWorld success rate by 14.1-18.8% and WebShop accuracy by 11.8-19.0% over RL baselines, and the student surpasses its own skill-conditioned teacher in every tested setting. The authors note that privileged task skills alone do not guarantee a reliable distillation teacher in agentic tasks. Both sources (Hugging Face daily papers, 2026-09-16; arXiv cs.AI/cs.LG/cs.CL, 2026-09-17) report the same method and figures with no discrepancies.

• ALFWorld success rate improves 14.1-18.8% over RL baselines.
• WebShop accuracy improves 11.8-19.0% over RL baselines.
• Tested on Qwen2.5 models from 1.5B to 7B across ALFWorld and WebShop multi-turn agent tasks.
• Two-stage method: a decoupled, skill-conditioned teacher is trained with environment rewards, then a skill-free student is trained jointly with RL and on-policy distillation.
• Dense token-level supervision comes from a self-teacher combined with RL.
• Adaptive Retirement drops the teacher once the student-teacher discrepancy stops shrinking and the student reaches a target fraction of the teacher's success rate.
• The student surpasses its own skill-conditioned teacher in every tested setting.
• Privileged task skills alone do not guarantee a reliable distillation teacher in agentic tasks.