RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
RetireOPD adds self-retiring on-policy distillation to agentic RL, boosting Qwen2.5 agent success on ALFWorld by 14-19% and WebShop by 12-19%.
RetireOPD first optimizes a decoupled, skill-conditioned teacher with environment rewards, then trains a skill-free student jointly with RL and on-policy distillation, retiring the teacher adaptively once student-teacher discrepancy stops shrinking. Across Qwen2.5 models from 1.5B to 7B, it improves ALFWorld success rate by 14.1-18.8% and WebShop accuracy by 11.8-19.0% over RL baselines. The student surpasses its own skill-conditioned teacher in every tested setting.
- Privileged task skills alone do not guarantee a reliable distillation teacher in agentic tasks.
- Adaptive Retirement drops the teacher once discrepancy stops shrinking at a target success fraction.
- ALFWorld success improves 14.1-18.8% and WebShop accuracy 11.8-19.0% over RL baselines on Qwen2.5 1.5B-7B.
- Student surpasses its own skill-conditioned teacher in every tested setting.
- AI models
- Qwen2.5
From the original article
1 min read at huggingface.co
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them.
By Yan Yu, Zhengxi Lu, Yizhou Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.