Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

ActObs adds supervision on environment-observation tokens during SFT — with no extra data, parameters, tokens, or forward passes — and GRPO initialized from ActObs outperforms action-only SFT on Terminal-Bench 2.0 for Qwen3-4B at every sampling budget, with…

The paper introduces ActObs, an SFT variant that applies prediction loss to environment-observation tokens already present in agent trajectories, in addition to the usual supervision on action tokens. On Qwen3-4B, GRPO initialized from ActObs achieves higher pass@k than action-only SFT at every sampling budget on Terminal-Bench 2.0. On Qwen3-8B, the method trades some pass@1 reliability for a +3.4 pp gain at pass@16 while solving more distinct tasks. The benefit also transfers to unseen code-editing tasks: ActObs yields +4.2 pp pass@1 on aider-polyglot at the 4B scale. The authors' gradient analysis traces the advantage to action and observation gradients becoming orthogonal under joint supervision, which preserves environment (consequence) prediction and policy entropy, improving exploration during downstream RL. Both source reports agree on all reported figures.

• ActObs supervises environment-observation tokens already in agent trajectories during SFT, requiring no extra data, parameters, tokens, or forward passes.
• Qwen3-4B: GRPO initialized from ActObs achieves higher pass@k at every sampling budget on Terminal-Bench 2.0 compared with action-only SFT.
• Qwen3-8B: +3.4 pp pass@16, at the cost of some pass@1 reliability, while solving more distinct tasks.
• Cross-domain transfer: +4.2 pp pass@1 on unseen aider-polyglot code-editing tasks at the 4B scale.
• Gradient analysis shows action and observation gradients become orthogonal, preserving environment prediction and policy entropy to support downstream RL exploration.