Paper treats long-context retrieval as value-indexed local traversal
Original title: Retrieval as Traversal: Value-Indexed Memory for Stateful Long-Context Agents
On MiniGrid Memory, a five-token traversal head reached 0.95 mean return after 0.59M steps, ahead of absolute addressing.
The paper proposes persistent local traversal through an address space, separating memory content from address. A cost-regularised index keeps high-value memories locally reachable and pushes redundant ones apart. Only the simplest fixed-geometry case was tested on MiniGrid Memory; learned geometry and repeated retrieval were not evaluated.
Unconfirmed: The experiment does not test learned geometry or repeated retrieval.
- The traversal head used a five-token context and reached 0.95 mean return.
- Training took 0.59M steps versus 0.94M for Gaussian absolute addressing.
- Categorical absolute addressing needed more than 2M steps.
- The authors say further experiments are required for the full method.
From the original article
1 min read at arxiv.orgLong-context models repeatedly solve closely related retrieval problems. Attention or global retrieval can retain large stores, but each forward pass again searches for what matters. Recurrent compression avoids this search only by deciding earlier what information to preserve.
By Daniel Strauß · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.