Tokens Too Cheap to Meter
Analysis argues LLM inference cost per task is falling by orders of magnitude yearly through GPU, model, and inference-engine efficiency gains.
The post surveys evidence that AI cost per task is dropping by orders of magnitude annually: GPU power efficiency roughly doubles every two years, per-task costs fell about two orders of magnitude between 2025 and 2026 frontier charts, and inference engines like vLLM improved roughly 40% in energy efficiency over 15 months. Mixture-of-Experts architectures and serving-stack advances compound these gains. The author predicts LLMs become pervasive infrastructure within one to two years and run at frontier quality locally on commodity hardware within 3-6 years.
- Cost per task fell roughly two orders of magnitude from 2025 to 2026
- GPU efficiency doubling about every two years, echoing Moore's Law pace
- vLLM ~40% more energy-efficient in 15 months; serving gains 10-50% yearly
- Predicts local frontier-quality models on commodity hardware within 3-6 years
From the original article
14 min read at jyn.devThe price of using machine learning intelligence is decreasing by several orders of magnitude a year and shows no signs of slowing. We are likely to see LLMs integrated into every part of computing as infrastructure, not just as a product, in the next year or two. We are likely to see LLMs running locally at current frontier-quality on commodity hardware in the next 3-6 years.
By teoruiz · jyn.dev
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.