Cortex: Consolidating Context into Parametric Memory
Cortex writes context gains into model weights, recovering 88.9% after 50 edits on a 14B model.
Cortex treats memory consolidation as internalizing context-driven gains while preserving unrelated capabilities, covering editing, skill learning, and behavioral steering. It generates prospective queries and distills context-conditioned outputs into mergeable low-rank or structural feed-forward updates. Across 4B–120B dense and mixture-of-experts backbones, including multimodal models, it sits on the consolidation–retention Pareto frontier in most settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain and retains 99.8% of non-target performance.
- Cortex distills context-conditioned behavior into mergeable low-rank or FFN updates.
- It is tested on dense and MoE models from 4B to 120B parameters.
- After 50 sequential factual edits on a 14B model, Cortex-Structural keeps 88.9% of the gain.
- Non-target performance retention is 99.8% in that setting.
From the original article
1 min read at huggingface.co
Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective.
By Jintian Zhang, Feng He, Xinnian Liang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.