Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes
Independent developer Hugo Vergnes trained a 3.8B-parameter Llama-style model to 0.384 CORE on 65B tokens for $998 in 43 hours on rented B200s.
Hugo Vergnes trained little-lm, a 3.848B-parameter decoder-only LLM, on 65.3B tokens in 43 hours for $998 using rented NVIDIA B200s, scoring 0.384 on the CORE benchmark and beating nanochat d32 (0.310) at similar cost. The Llama-style architecture uses RMSNorm, RoPE, GQA with 24 query and 8 KV heads, relu-squared MLPs, QK-norm, and ResFormer-style value embeddings that account for 19% of parameters. Key wins included the Muon optimizer for matrix parameters, a trapezoidal learning-rate schedule with linear cooldown, FP8 training plus vocabulary padding for roughly 33% throughput gains, and the ClimMix dataset over FineWeb-Edu. The project, inspired by Karpathy's nanochat, was built as a config-driven YAML framework for small LLM training.
- 3.8B model reached 0.384 CORE, ahead of nanochat d32's 0.310 at similar cost
- FP8 training and vocab padding delivered roughly 33% throughput gains
- Muon optimizer on matrix parameters converged faster than AdamW-only at this scale
- Trapezoidal LR schedule with linear cooldown kept eval loss falling through the final step
- Value embeddings make up 19% of total parameters across 14 tables
- Vendors
- NVIDIA
- Organizations
- NVIDIA
- AI models
- little-lm-3.8B
From the original article
15 min read at hugovergnes.github.ioSomewhere between “nanoGPT toy” and “you need a research lab” there’s a large, under-described region where one person with a few thousand dollars can train a meaningful model. I wanted to see language and understanding emerge from random weights for myself, and to learn the parts you can only learn by starting from scratch. This project was written in the evenings, debugged on a 5090 and finished on rented B200s.
By Anon84 · hugovergnes.github.io
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.