ConvMem: Convolutional Memory for Long-Context Reasoning
Researchers propose ConvMem, a training-free framework treating LLMs as convolutional kernels for parallelizable long-context reasoning beyond fixed context windows.
ConvMem reformulates long-context reasoning as a hierarchical convolution in which the LLM summarizes text segments hierarchically, shortening the reasoning path from a linear chain to a logarithmic tree. It uses configurable strides, skip connections, and multi-kernel convolution to capture evidence, decompose queries, and enable massive parallelization across segments and reasoning threads. On RULER-HotpotQA and RULER-2WikiMultiHopQA it outperforms training-free baselines and avoids the out-of-distribution overfitting seen in RL-trained approaches like MemAgent.
- Training-free and highly parallelizable long-context framework
- Treats the LLM prompted with a query as a convolutional kernel
- Shortens reasoning path from linear chain to logarithmic tree
- Outperforms training-free baselines on RULER-HotpotQA and RULER-2WikiMultiHopQA
- Avoids overfitting risk of RL-trained sequential memory methods
From the original article
1 min read at arxiv.orgWhile Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context limits. To address this, sequential approaches like MemAgent extend the effective context by reading text in segments and iteratively updating a fixed-size memory.
By Hongming Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.