Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models
Audit of 22 frontier models finds widespread verbatim retrieval of published molecular property values, with higher reasoning increasing recall of memorized numbers.
An arXiv audit tests 22 frontier LLMs across 12 molecular regression benchmarks for verbatim retrieval of published values. More than 50% of the LLMs show verbatim retrieval on five datasets, and identical experiments are flagged 89% more often at a high reasoning level than at the lowest one. Suppressing retrieval moves model prediction errors closer together in relative terms, suggesting predictive capability is not determined solely by memorized values.
- 22 frontier models audited on 12 molecular regression benchmarks for verbatim retrieval
- Over 50% of LLMs retrieve verbatim values on five datasets
- Higher reasoning level increases retrieval flags by 89% on identical prompts
- Suppressed retrieval brings model prediction errors closer together relatively
From the original article
1 min read at arxiv.orgLarge language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy cannot distinguish a model that predicts a property from one that retrieves a published number.
By Matthias Busch · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.