MemT-1.0 releases Nano and Flash memory-native models
Original title: MemT-1.0: Towards Memory-Native Evolution
At 128K history, Nano is about 1.86 times faster and uses 66 times less persistent storage than a full-context KV cache.
MemT-1.0 is a memory foundation-model architecture that internalizes memory in computation, building on Metis with Gated DeltaNet-2 and Mixture-of-Storage. A two-stage pipeline of continued pre-training and post-training uses 34B tokens. The released MemT-1.0-Nano and MemT-1.0-Flash beat parametric memory baselines on average and stayed competitive with retrieval-augmented methods while largely preserving general capability. Internalizing retrieved procedural experience improved success on agentic tasks.
Unconfirmed: Speed and accuracy figures are the authors' reported measurements.
- Training data totaled 34B tokens across continued pre-training and post-training.
- At 128K tokens, Nano reported about 1.86 times end-to-end inference speedup.
- Persistent storage fell 66 times versus a full-context KV cache.
- The released family is MemT-1.0-Nano and MemT-1.0-Flash.
From the original article
1 min read at huggingface.co
Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1. 0, a new architecture of memory foundation models that internalize memory as part of model computation.
By Zhiyu Li, Zeyu Zhang, Linyun Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.