RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
RetireOPD trains multi-turn agents via self on-policy distillation with adaptive teacher retirement, lifting Qwen2.5 ALFWorld success by up to 18.8%.
RetireOPD first trains a decoupled, skill-conditioned teacher with environment rewards, then trains a skill-free student jointly with reinforcement learning and on-policy distillation. Its Adaptive Retirement mechanism drops the teacher once their discrepancy stops shrinking and the student reaches a target fraction of the teacher's success rate. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success by 14.1-18.8% and WebShop accuracy by 11.8-19.0% over RL baselines, surpassing its own teacher in every setting.
- Dense token-level supervision from a self-teacher combined with RL
- Adaptive Retirement drops the teacher when discrepancy stops shrinking
- Tested on Qwen2.5 1.5B-7B across ALFWorld and WebShop tasks
- Student surpasses its own skill-conditioned teacher in all settings
- Organizations
- Alibaba
From the original article
1 min read at arxiv.orgMulti-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them.
By Yan Yu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.