ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
ScholarCatalyst asks models to retrieve papers that inspired 207 CS projects; agents fail to beat embedding retrieval.
ScholarCatalyst is a retrieval benchmark built from 184 lead authors of 207 recent computer-science papers who labeled which earlier papers did or could have advanced their projects, each with a rationale. Given an initial research question, systems must retrieve those papers from only the literature available when the project began. Agentic search scored 0.42 Recall@20 against 0.48 for embedding retrieval, and an agent built on Claude Fable 5.1 reached only 0.51 R@20.
- 184 lead authors labeled inspiring prior work for 207 recent computer-science papers.
- Systems must retrieve those papers from literature available when each project began.
- Agentic search scores 0.42 Recall@20 versus 0.48 for embedding retrieval.
- A Claude Fable 5.1 agent reaches only 0.51 Recall@20.
Full article211 words · extracted from arxiv.org · click to collapse
What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.02202