Low-bit n-gram embeddings cut table memory with little loss
Original title: Low-Bit N-Gram Embeddings for Language Models
Four-bit tables saved about 75% of table memory on models trained from scratch while keeping over 99% of the validation-loss gain.
The paper studies low-precision storage of n-gram embedding tables used by recent open-weight language models. On nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables saved approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from the tables. On DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables saved 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream performance. Reinvesting saved memory in larger tables is reported to beat the original full-precision tables at lower memory use.
- DeepSeek-V4.1-Flash is said to allocate 196 billion parameters to Engram n-gram memory.
- Four-bit tables preserved more than 99% of the validation-loss improvement.
- Fewer bits caused a larger drop that short fine-tuning partly recovered.
- Large-model savings were 47-55% of table memory with perplexity up under 1.2%.
From the original article
1 min read at arxiv.orgMany recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.
By Stephen Chung · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.