Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Researchers propose Feedback-Enriched Environments (FEEs) that reduce reward sparsity and improve RL training of Qwen3-based agents on SciWorld and BFCL.
The paper proposes shifting from agent-side warmup (SFT) to environment-side adaptation via Feedback-Enriched Environments to address severe reward sparsity in RL training of long-horizon LLM agents. A pilot study defines a feedback strategy that transitions from action guidance to observation enrichment in later training stages. Large-scale experiments on SciWorld and BFCL across Qwen3 model scales and GRPO, GSPO, and DAPO show consistent gains, plus stabilized training dynamics and proactive exploration.
- Feedback-Enriched Environments shift adaptation from agent-side SFT to environment-side feedback during exploration and evolution.
- Consistent gains on SciWorld and BFCL across Qwen3 scales with GRPO, GSPO, and DAPO.
- Reduces entropy volatility, stabilizing RL training dynamics on hard long-horizon tasks.
- Intra-group feedback consistency identified as a boundary condition for stable optimization.
- AI models
- Qwen3
From the original article
1 min read at huggingface.co
Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional agent-side warming up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration.
By Hongbang Yuan, Zhuoran Jin, Yixin Cao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.