SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning
SERO trains long-horizon agents using executable oracle continuations and reachability weighting on ALFWorld and WebShop.
SERO augments on-policy agentic reinforcement learning with executable oracle continuations from states the current policy actually reaches, so downstream observations and oracle actions can supervise unvisited futures. Each oracle step is weighted by the frozen behavior policy's compatibility with preceding oracle actions, and the oracle loss is optimized separately from the base RL estimator. The oracle is training-only and is not used at inference. On ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, full continuations outperform root-action supervision, and reachability weighting improves out-of-distribution results.
- SERO lets a training-only oracle continue from states the current policy actually reaches.
- Policy-reachability weighting scales oracle-step supervision by the frozen behavior policy's compatibility.
- Full executable continuations beat supervising only the root oracle action on ALFWorld.
- No oracle is used at inference; the oracle loss is separate from the base RL estimator.
From the original article
1 min read at huggingface.co
Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered.
By Dingyi Rong, Yue Shi, Chaofan Ma · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.