Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
ActObs supervises observation tokens during SFT, improving GRPO-trained agent pass@k on Terminal-Bench 2.0 and cross-domain code editing for Qwen3 models.
The paper introduces ActObs, an SFT variant that adds prediction loss on environment observation tokens already present in agent trajectories. After GRPO on Qwen3-4B, ActObs achieves higher pass@k at every sampling budget on Terminal-Bench 2.0 than action-only SFT. On Qwen3-8B it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and improves cross-domain code editing on aider-polyglot (+4.2 pp pass@1 at 4B). Analysis shows action and observation gradients become orthogonal, preserving environment prediction and exploration for downstream RL.
- ActObs supervises observation tokens already in agent trajectories without extra data, parameters, tokens, or forward passes.
- GRPO from ActObs beats action-only SFT in pass@k at every budget on Terminal-Bench 2.0 (Qwen3-4B).
- On Qwen3-8B it gains +3.4 pp pass@16 while trading some pass@1 reliability.
- Cross-domain code editing on aider-polyglot improves +4.2 pp pass@1 at 4B on unseen tasks.
- Action and observation gradients become orthogonal, preserving consequence prediction and entropy during RL.
- Vendors
- Qwen
From the original article
1 min read at arxiv.orgAgent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning.
By Juzheng Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.