LatComm compresses LLM context once for reuse across models
Original title: LatComm: Reusable Context Compression for Latent Communication between Large Language Models
On QuALITY the receiver sees 57 times less input and answers an article's questions three times faster.
LatComm encodes a shared context once, without the question, using a frozen LLM into a fixed-size packet. A different frozen receiver reads that packet through gated cross-attention for each question. Across 12 multiple-choice QA datasets it improves average accuracy over single-vector baselines by up to 10.2 points. On QuALITY it reduces the receiver's input by 57x and answers an article's questions 3x faster.
- Gates that choose what to read are computed from the receiver's prompt.
- Encoding is separated from reading so one packet serves many questions.
- The design targets compact, reusable context that a different LLM can read.
From the original article
1 min read at arxiv.orgSystems that combine several large language models (LLMs) often use one context for many questions. When the context is passed between models as text, the receiving model processes all of its tokens again for every question. Avoiding this cost requires a context representation that is compact, reusable across questions, and readable by a different LLM.
By Huawei Lin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.