Co-Evolving Acting and Forecasting for Language Agents
CoPE trains language agents to act and forecast future actions, improving benchmark scores and sample efficiency.
The paper proposes CoPE, a training method that co-evolves acting and forecasting for language agents by supervising a shared policy backbone with multi-step action targets from successful on-policy trajectories. Unlike value or next-observation auxiliary targets, each action is also trained when forecast from earlier histories, while inference remains single-step with no added model or latency. Across ALFWorld, ScienceWorld, WebShop, tau-squared-bench, and AppWorld, the authors report stronger aggregate results than GRPO and RL baselines that use predictive auxiliary targets. On ScienceWorld, they report up to 1.9x and 2.0x higher sample efficiency than ECHO and GRPO.
- CoPE adds multi-step action-forecast targets to on-policy language-agent training.
- Inference still takes one action at a time with no extra model or overhead.
- Gains are reported on ALFWorld, ScienceWorld, WebShop, tau2-bench, and AppWorld.
- On ScienceWorld, sample efficiency is up to 1.9x ECHO and 2.0x GRPO.
- Organizations
- University of Illinois Urbana-Champaign
From the original article
1 min read at arxiv.orgTraining language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response.
By Haofei Yu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.