REMORY: Learning Residual Memory for Context Compaction
REMORY adds residual soft memory tokens so agents compact context without losing later decision quality.
REMORY is a neural memory network that supplements a textual summary with a bounded sequence of soft memory tokens. Conditioned on the history and summary, the tokens help a frozen LLM approximate the continuation it would produce from the full history. On SummHay it improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using 5.2% of input positions. Qwen3.8-27B and GLM-5.3-Flash gain on long-horizon agent benchmarks and make fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.
- Soft memory tokens act like a residual connection after the summary.
- SummHay approaches full-context score using 5.2% of positions.
- Qwen3.8-27B and GLM-5.3-Flash show consistent agent-benchmark gains.
- Both models cut repeated tool outputs and errors on BrowseComp and Terminal-Bench 2.1.
Full article144 words · extracted from huggingface.co · click to collapse
Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the history and summary, the network learns to generate tokens that help a frozen LLM approximate the continuation it would produce with the full history. The tokens are conditioned on the summary and appended after it, forming an analogue of a residual connection along the sequence dimension. On SummHay, REMORY improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using only 5.2% of the input positions. Across long-horizon agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also exhibit substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.11287