Deep Fusion Memory adds chunk-level parametric memory to LLMs
Original title: Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models
With the backbone frozen, training needs 32 times fewer retrieval queries and inference uses no explicit retrieval.
Deep Fusion Memory predicts chunk embeddings from preceding context and fuses them into hidden states with gated cross-attention, sharing one memory across tokens in a chunk. Prediction and fusion train end to end while the language-model backbone stays frozen. On six QA tasks the paper reports gains of 12.71 percent over the strongest baseline on Mistral-7B-v0.3 and 11.19 percent on Qwen3-8B-Base.
- Construction uses 2048 times fewer retrieved items than token-level targets.
- Domain adaptation adds 6.51 points on Qwen3-4B-Base.
- The same adaptation adds 4.38 points on Qwen3-8B-Base.
- Inference does not call an online retriever.
From the original article
1 min read at huggingface.co
Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead.
By Rubin Wei, Siyuan Sun, Hao Sun · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.