Ted Sanders says model size explains most LLM performance gaps
Original title: Quoting Ted Sanders
Bigger models generally cost more per token and recall obscure facts better, even when skill rankings differ.
Simon Willison quotes Ted Sanders on LLM performance being multidimensional, with no fixed ranking across tasks such as math and jokes. Sanders says model size still explains much of the variation: larger models are generally more capable, cost more per token, and recall esoteric facts better. He orders models as Luna, then Terra, then Sol, then Astra, and begins explaining what version numbers signify.
Unconfirmed: The quote is truncated and does not name the company behind Luna, Terra, Sol, and Astra.
- Sanders says larger models are more reliable and cost more per token.
- The stated model order is Luna, Terra, Sol, then Astra.
- Version numbers are said to reflect release order and improvement size.
From the original article
In general, LLM performance is incredibly multidimensional. There's no guarantee of a fixed ordering where the model that is best at math is also the model that's best at jokes. Despite that complexity, much of the variation in performance is explained by just a single variable: model size.
simonwillison.net
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.