Latent Compaction with Test-Time Training
Latent Compaction preserves long-context retrieval at 32x KV-cache compression via test-time training.
Latent Compaction compresses long-context key-value caches by treating the cache as a neural network and optimizing it with test-time training while model weights remain frozen. Cache entries are initialized using reconstruction importance, teacher attention, and key-space diversity, then distilled from the full-cache model without access to downstream questions. On RULER the method reaches 96.7% accuracy at 32x compression, compared with 40.0% for KVZip+, and it also improves retrieval on LongBench and Phonebook Lookup across Qwen3 and MiniCPM5.
- Latent Compaction learns a document-specific compact KV cache at test time.
- Model weights stay frozen and training uses only generic document prompts.
- On RULER it reaches 96.7% accuracy at 32x compression versus 40.0% for KVZip+.
- Retrieval gains transfer to Qwen3 and MiniCPM5.
- Code is released by THUNLP.
- Organizations
- THUNLP
From the original article
1 min read at arxiv.orgLong contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce **Latent Compaction**, which learns a compact document-specific cache through test-time training.
By Yingfa Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.