OpenDS-World tests deep search agents on the live Web
Original title: Deep Search in Real-World Web Environments
The best of 12 frontier models scores 39.0%; humans score 21.1% across 1,000 attempts.
The paper introduces OpenDS, a task for multi-step deep search on the live Web, and OpenDS-World, a 200-task benchmark with human-verified evidence and unique answers. Agents must search, browse, and integrate evidence from independent sources. Twelve frontier models were evaluated under one tool environment and protocol. Analyses of tool use, cost, termination, and failure recovery show persistent coordination and long-horizon search problems.
Unconfirmed: The paper does not name which of the 12 models achieved 39.0% accuracy.
- OpenDS-World contains 200 tasks with independent human verification.
- Tasks span modalities, file formats, and interactive access requirements.
- Analyses cover tool usage, search cost, termination, and failure recovery.
- The paper does not name the model that reached 39.0% accuracy.
From the original article
1 min read at huggingface.co
Despite substantial progress in Deep Search agents, their ability to conduct end-to-end research in real-world open Web environments remains insufficiently evaluated.
By Xiaomin Yu, Hongliang Chen, Zhe Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.