LLM agents reproduce 77.4% of claims in SEER cancer papers
Original title: AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery
A robustness pipeline flags potentially consequential analytical choices in 9.5% of fully reproduced findings for expert review.
Researchers test whether LLM agents can automate reproduction and robustness checks for studies based on the U.S. SEER cancer registry. SEER-Repro contains 20 published papers plus the data and resources needed to reproduce them. One agent reproduced 77.4% of 2,190 atomic claims and supplied executable baselines. A second pipeline proposes alternative analytical choices, reruns the analysis, and sends potentially sensitive findings to expert review.
- SEER-Repro covers 20 published papers based on the SEER registry.
- The reproduction agent evaluated 2,190 atomic claims.
- The robustness pipeline identifies analytical choices and reruns the analysis.
- Flagged findings are sent for expert review rather than automatically overturned.
- Products
- SEER
- Countries
- United States
From the original article
1 min read at arxiv.orgReproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U. S.
By Shicheng Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.