Stanford study finds 44% of LLM claims expired
Original title: Rapid LLM development is outrunning the research meant to explain it, study finds
Stanford researchers tested 50 LLM claims and found 44% expired after an average of 15.7 months.
Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models released between July 2024 and September 2026. They found 44 percent have already expired and 34 percent have weakened significantly. Only 16 percent still hold. Claims about what models cannot do expire fastest, while some biases and sycophancy persist longer. The study concludes empirical findings about LLMs should be presented as observations about specific models rather than general truths.
- Average lifespan of expired findings is 15.7 months
- All nine reasoning-related findings have expired
- Hallucinated references dropped from 55% to 0% in GPT-3.5 but remain high in some 2026 models
- Biases vary: left-leaning political bias persists, while some gender stereotypes weakened
- Organizations
- Stanford University
From the original article
3 min read at the-decoder.com
Stanford researchers tested 50 widely cited claims about LLMs against 20 current models. 44 percent of the findings are already "expired," and another 34 percent have weakened significantly. The average lifespan is just 15.
By Matthias Bastian · the-decoder.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.