SearchHorizonBench tests whether AI agents can value another search
Original title: SearchHorizonBench: A Benchmark for Search Value in AI Agents
On 1,632 held-out questions, historical calibration cut Brier error 16.2%, but point estimates explain under 2% of search-gain variance.
SearchHorizonBench evaluates whether AI agents can estimate the extra benefit of another search from execution experience, rather than only question difficulty. The resource has 45,632 executions across 3,168 questions, with paired continuations and repeated runs separating expected search gain from generation noise. On 1,632 held-out questions, historical calibration reduces Brier error by 16.2%, mostly through success-level information. At the reference token price, free selection observations yield a 2.49-point utility opportunity over the best constant action.
- Search gains have a standard deviation of 21.4 percentage points.
- Tested historical estimators explain less than 2% of that variance.
- The gain over the best constant action is driven mainly by gain information.
- Reusable histories, frozen splits, baselines and scoring tools are included.
From the original article
1 min read at huggingface.co
AI agents must decide when another search is worth its cost. Answer-success probabilities capture question difficulty, while this decision requires estimating the additional benefit of search. We introduce SearchHorizonBench, a benchmark for learning and evaluating search value from execution experience.
By Jingjie Ning, Xueqi Li, Yibo Kong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.