Richard Sutton warns LLM scaling enthusiasts
Original title: The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts
Sutton calls for world-model agents with continuous learning to avoid forgetting.
Turing Award winner Richard Sutton argues in a recent post that raw compute scaling is insufficient for intelligence and calls for new learning methods. He criticizes synthetic data as a big mistake and advocates agents that learn from experience without losing old knowledge. Other researchers like Jerry Tworek and Jayden Teoh discuss the need to explore scaling dimensions beyond transformers. Industry leaders Sam Altman and Dario Amodei counter that LLM scaling is far from exhausted.
Unconfirmed: Altman and Amodei may be biased as CEOs of scaling-focused companies
- Sutton says synthetic data is a big mistake
- Needs continuous learning without destroying knowledge
- 45 new AI labs tracked betting on gaps
- World models enable post-training learning
- Scaling algorithms can hit dead ends
From the original article
3 min read at the-decoder.com
Turing Award winner Richard Sutton reaffirms that raw compute alone isn't enough for intelligence and that fundamentally new learning methods are needed. Ex-OpenAI researcher Tworek agrees, saying blind transformer scaling "has reward hacked brains of a whole generation of scientists." A growing wave of startups is already betting on exactly this gap.
By Matthias Bastian · the-decoder.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.