FOCUS uses selective branching to train long-horizon agent critics
Original title: FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
At Qwen3.5-9B, held-out accuracy rises 7.2 points on BrowseComp-Plus and 4.1 on DeonticBench versus the strongest baseline.
FOCUS is a reinforcement-learning method for long-horizon language-model agents that improves credit assignment when outcome rewards are sparse. A rollout controller samples continuations from states expected to most reduce critic error, then uses TD(λ) advantages and those samples as critic targets. Context management is separated from rollout allocation. With matched per-step generation budgets, the paper says FOCUS beats GRPO and VinePPO on legal reasoning, deep search, and code localization, including gains at Qwen3.5-9B scale.
- BrowseComp-Plus held-out accuracy is 55.4 versus 48.2 for the strongest baseline.
- DeonticBench held-out accuracy is 56.5 versus 52.4.
- Comparisons use matched per-step generation budgets against GRPO and VinePPO.
- Compaction-induced segment boundaries are used for value estimation.
From the original article
1 min read at arxiv.orgSparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a trajectory incurs substantial costs.
By Zhengping Jiang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.