ACCT transfers KV cache between LLMs for context reuse
Original title: ACCT: Asymmetric Cross-LLM Context Transfer
ACCT reduces KV cache size by 75% while matching target model performance and speeding up prefill by 21x.
LLM systems increasingly use multiple heterogeneous models for inference, but reusing the same context across them incurs repeated computational costs. ACCT transforms source KV cache into sparse states consumable by target models of different scales. The approach includes the Cartographer for query-relevant token selection, Allocator for KV budget distribution, and Compiler for transformation. Evaluations on general benchmarks and context-dependent tasks show matching or exceeding original performance with 75% KV reduction and 6.5-21.2x speedup at 19K tokens.
- ACCT matches target model performance at 75% KV reduction
- Speedup of 6.5-21.2x over target prefill at 19K context
- Three lightweight components for KV transfer
- Works within same model family
- Validates on five general and six context tasks
From the original article
1 min read at arxiv.orgIn real-world deployments, large language model (LLM) systems are increasingly evolving from single-model invocation toward heterogeneous deployments in which different models are dynamically invoked for inference. When the same context is reused across models, however, each newly selected target model typically has to process it again, repeatedly incurring substantial computational costs.
By Yuzhe Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.