MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup
MoME introduces context-aware memory embeddings with gated multi-slot lookup, improving LLM efficiency across Llama-3 and Qwen3 backbones.
Mixture of Memory Embeddings (MoME) replaces each token's single memory row with a mixture of M slots selected by a learned gate over the hidden state, resolving sense-collision in deterministic lookups. In controlled pretraining on nanochat, Llama-3/MobileLLM, and Qwen3 backbones, MoME beats Value Embedding, Bigram, and STEM baselines in iso-parameter and iso-FLOP settings. Routing analyses show semantic interpretability, dispatching polysemous tokens to distinct slots by context.
- Replaces deterministic memory lookup with gated mixture of M slots
- Evaluated on nanochat, Llama-3/MobileLLM, and Qwen3 backbones
- Beats Value Embedding, Bigram, and STEM baselines at matched compute
- Learned routing shows interpretable sense-based token dispatch
- Products
- nanochat
From the original article
1 min read at huggingface.co
Scaling large language models efficiently has motivated sparse capacity mechanisms such as Mixture-of-Experts and, more recently, conditional memory: token-indexed embedding tables that augment the backbone with cheap parametric lookups. Existing memory-embedding methods retrieve via a deterministic function of the surface form, which collapses different contextual senses of the same token (e. g.
By Muchen Li, Leonid Sigal, Renjie Liao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.