Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
ActObs adds observation-token supervision to SFT, yielding higher pass@k for Qwen3 agents after GRPO on Terminal-Bench 2.0 and code editing.
Researchers introduce ActObs, an SFT variant that supervises environment-observation tokens in agent trajectories in addition to action tokens, without extra data, parameters, tokens, or forward passes. On Qwen3-4B, GRPO initialized from ActObs achieves higher pass@k at every sampling budget on Terminal-Bench 2.0, and on Qwen3-8B it trades some pass@1 for +3.4 pp at pass@16 while solving more distinct tasks. The benefit transfers to unseen code-editing tasks on aider-polyglot (+4.2 pp pass@1 at 4B scale). The authors trace the advantage to gradient analysis showing joint supervision preserves environment prediction and policy entropy, improving downstream RL exploration.
- Supervising observation tokens during SFT improves RL readiness.
- Qwen3-4B gains on Terminal-Bench 2.0 at every sampling budget.
- Qwen3-8B: +3.4 pp pass@16, more distinct tasks solved.
- Cross-domain transfer: +4.2 pp pass@1 on aider-polyglot at 4B.
- Joint supervision preserves entropy and environment prediction.
From the original article
1 min read at huggingface.co
Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning.
By Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.