OpenAI publishes pretraining perplexity to anchor shared safety standards
Original title: Perplexity and Intelligence
The post cites log-perplexity of about 0.99 for GPT-3.5, 0.91 for GPT-4, and below 0.8 for newer models.
An OpenAI post argues that pretraining progress is best tracked by perplexity, or cross-entropy loss, and that the metric should compare developers and inform shared safety standards. It says internal measurements from GPT-3.5 onward show capabilities emerging as loss falls, including fluent chat near 0.99, chain-of-thought near 0.91, and cybersecurity and mathematics below 0.8. A model near 0.76 is said to have produced notable mathematics results. The author expects further changes in a roughly 0.68–0.7 window and says algorithmic pretraining advances could arrive as soon as 2027.
Unconfirmed: Capability thresholds, the 0.68–0.7 window, and a 2027 pretraining avalanche are the author's extrapolations, not demonstrated results.
- GPT-3.5 is given at about 0.99 log-perplexity and GPT-4 at about 0.91.
- Losses below 0.8 are tied to cybersecurity and mathematics capabilities.
- A model near 0.76 is said to have resolved long-standing math questions.
- The author flags a possible step change from AI doing pretraining research.
- Organizations
- OpenAI
From the original article
6 min read at openai.com
How perplexity measures progress in AI models, and why it could help developers establish a shared approach to evaluating emerging capabilities.
openai.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.