Recreating Minecraft Is Not a Benchmark
Opinion piece argues viral demos like one-prompt Minecraft recreations are overfit 'demo-benchmarks' measuring preparation, not true model capability.
The author argues that fixed, famous demo tasks (Minecraft builds, SVG pelicans) are trivially optimizable by labs each release cycle, so they no longer differentiate model capability. The piece cites Thinking Machines' Inkling Small scoring within a point of its flagship on the Artificial Analysis Intelligence Index with less than a third of the parameters, and beating it on Humanity's Last Exam, GPQA Diamond, and SciCode. The proposed alternative is rotating or holdout evals such as LiveBench, ARC-AGI's private set, and held-back portions of Humanity's Last Exam.
- Static demo-benchmarks leak into training data and can be overfit on schedule.
- Inkling Small nearly matched its flagship on AAII with under a third of the parameters.
- Holdout and rotating evals (LiveBench, ARC-AGI private set) resist teaching-to-the-test.
- Demos remain valuable for communication even though they are weak benchmarks.
- Vendors
- Thinking Machines
- Organizations
- Thinking MachinesArtificial Analysis
- AI models
- GPT AstraInkling Small
From the original article
2 min read at kuber.studio
GPT Astra released a couple of days ago and, inevitably, within the hour my entire feed was the same five things: recreating Minecraft in one prompt, painting themselves in MS Paint, the pelican riding a bicycle as an SVG, a ball bouncing in a rotating box with believable gravity, and an SVG game controller.
By kuberwastaken · kuber.studio
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.