RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
RetireOPD adds self-retiring on-policy distillation to agentic RL, boosting Qwen2.5 agent success on ALFWorld by 14-19% and WebShop by 12-19%.
RetireOPD first optimizes a decoupled, skill-conditioned teacher with environment rewards, then trains a skill-free student jointly with RL and on-policy distillation, retiring the teacher adaptively once student-teacher discrepancy stops shrinking. Across Qwen2.5 models from 1.5B to 7B, it improves ALFWorld success rate by 14.1-18.8% and WebShop accuracy by 11.8-19.0% over RL baselines. The student surpasses its own skill-conditioned teacher in every tested setting.
- Privileged task skills alone do not guarantee a reliable distillation teacher in agentic tasks.
- Adaptive Retirement drops the teacher once discrepancy stops shrinking at a target success fraction.
- ALFWorld success improves 14.1-18.8% and WebShop accuracy 11.8-19.0% over RL baselines on Qwen2.5 1.5B-7B.
- Student surpasses its own skill-conditioned teacher in every tested setting.
Full article172 words · extracted from huggingface.co · click to collapse
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.20784