nanochat (Product): news timeline & CVEs

MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

MoME introduces context-aware memory embeddings with gated multi-slot lookup, improving LLM efficiency across Llama-3 and Qwen3 backbones.

Mixture of Memory Embeddings (MoME) replaces each token's single memory row with a mixture of M slots selected by a learned gate over the hidden state, resolving sense-collision in deterministic lookups. In controlled pretraining on nanochat, Llama-3/MobileLLM, and Qwen3 backbones, MoME beats Value Embedding, Bigram, and STEM baselines in iso-parameter and iso-FLOP settings. Routing analyses show semantic interpretability, dispatching polysemous tokens to distinct slots by context.

Hugging Face daily papersSep 13, 20:00 UTC · 13d agoAI research2

Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes

Independent developer Hugo Vergnes trained a 3.8B-parameter Llama-style model to 0.384 CORE on 65B tokens for $998 in 43 hours on rented B200s.

Hugo Vergnes trained little-lm, a 3.848B-parameter decoder-only LLM, on 65.3B tokens in 43 hours for $998 using rented NVIDIA B200s, scoring 0.384 on the CORE benchmark and beating nanochat d32 (0.310) at similar cost. The Llama-style architecture uses RMSNorm, RoPE, GQA with 24 query and 8 KV heads, relu-squared MLPs, QK-norm, and ResFormer-style value embeddings that account for 19% of parameters. Key wins included the Muon optimizer for matrix parameters, a trapezoidal learning-rate schedule with linear cooldown, FP8 training plus vocabulary padding for roughly 33% throughput gains, and the ClimMix dataset over FineWeb-Edu. The project, inspired by Karpathy's nanochat, was built as a config-driven YAML framework for small LLM training.

Hacker News · securitySep 10, 02:04 UTC · 17d agoAI research HN 28↑ · 2 comments1

nanochat

Timeline2 articles

Appears with

MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes