Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
ActObs adds observation-token supervision to SFT, yielding higher pass@k for Qwen3 agents after GRPO on Terminal-Bench 2.0 and code editing.
Researchers introduce ActObs, an SFT variant that supervises environment-observation tokens in agent trajectories in addition to action tokens, without extra data, parameters, tokens, or forward passes. On Qwen3-4B, GRPO initialized from ActObs achieves higher pass@k at every sampling budget on Terminal-Bench 2.0, and on Qwen3-8B it trades some pass@1 for +3.4 pp at pass@16 while solving more distinct tasks. The benefit transfers to unseen code-editing tasks on aider-polyglot (+4.2 pp pass@1 at 4B scale). The authors trace the advantage to gradient analysis showing joint supervision preserves environment prediction and policy entropy, improving downstream RL exploration.
- Supervising observation tokens during SFT improves RL readiness.
- Qwen3-4B gains on Terminal-Bench 2.0 at every sampling budget.
- Qwen3-8B: +3.4 pp pass@16, more distinct tasks solved.
- Cross-domain transfer: +4.2 pp pass@1 on aider-polyglot at 4B.
- Joint supervision preserves entropy and environment prediction.
Full article225 words · extracted from huggingface.co · click to collapse
Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.20715