EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory
EngramEdit updates LLM facts in conditional memory while keeping the Transformer backbone fixed.
EngramEdit updates factual knowledge stored in conditional n-gram memory, such as DeepSeek Engram, without changing the Transformer backbone. It computes target memory representations so an updated fact is predicted across multiple expressions, then jointly edits shared embeddings while penalizing changes to frequently reused ones. The paper reports near-perfect editing success, use of revised facts on unseen expressions and in multi-hop reasoning, and nearly three times the strongest baseline's accuracy under chain-of-thought prompting. Unrelated knowledge and general capabilities are largely preserved as edits accumulate.
- EngramEdit edits n-gram memory while the Transformer stays fixed.
- Shared embeddings are updated jointly and frequent ones are penalized.
- Near-perfect edit success and about 3x baseline CoT accuracy.
- Unrelated knowledge and general capabilities remain largely intact.
- Organizations
- DeepSeek
- AI models
- DeepSeek Engram
From the original article
1 min read at arxiv.orgConditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation.
By Hongru Cai · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.