Frozen 16 MB codes replace learned embeddings in small models
Original title: Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
A Gemma-shaped ternary model fits in a 46 MB file and runs 615–740 tokens per second on a laptop CPU.
The paper replaces a small language model's learned embedding table with one frozen 16 MB table of 512-bit codes and no trainable parameters, derived once by principal-component projection and sign binarisation. A Gemma-shaped body trained from scratch on 2.05B tokens with the frozen table matches or beats the learned table on LAMBADA, HellaSwag and PIQA; only ARC-Easy favours the learned table. With a ternary BitNet b1.58 body the model is 46 MB, ties the float model on LAMBADA and PIQA, and is five to six times faster than the float model's 8-bit GGUF in llama.cpp at a seventh of the memory. Weights, data and evaluation answers are released.
- In Gemma 3 270M the embedding table holds 168M of 268M parameters.
- The learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters.
- The frozen table wins by 1.7 to 2.3 nats on rare words.
- It accepts 10,000 new words after training at a 3% cost; the learned table drops 43%.
- Laptop generation uses 62 MB of memory at 615 to 740 tokens per second.
- Products
- llama.cpp
- AI models
- Gemma 3 270MBitNet b1.58
From the original article
1 min read at huggingface.co
In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys.
By Sai Kiran Bathula · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.