MemoryAthena: Adaptive Routing over Latent and Generated Memories
MemoryAthena routes generated memory as a bounded correction to direct Engram retrieval.
MemoryAthena tests whether useful memory can be generated as well as retrieved, using direct Engram retrieval, generation from retrieved Engram cues, and generation from causal backbone states. With the backbone, memory, generators, and readers frozen, a lightweight causal routing head learns when generated memory should modify the direct-retrieval residual through bounded interpolation. Question-answering performance rises from 37.65 to 39.28 on a five-task average, while a six-task general-NLP average rises from 76.73 to 79.13. The complete memory-side system contains about 201 million parameters, excluding the frozen backbone.
- Pathways cover direct retrieval and two generation modes
- A frozen-backbone routing head uses counterfactual token advantages
- Five-task QA average rises from 37.65 to 39.28
- Memory-side system contains about 201 million parameters
Full article225 words · extracted from huggingface.co · click to collapse
Learned-memory methods store information in an explicit table and consume it through a separate reader, allowing addressing, storage, and reading to be modified independently. We study whether useful memory can also be generated rather than only retrieved. MemoryAthena uses three pathways: direct Engram retrieval (E), generation from retrieved Engram cues (GE), and generation from causal backbone states without consulting the memory table (GH). Generated memory is conditionally useful: it can complement E in one context but interfere with it in another. MemoryAthena therefore treats E as an anchor and learns when a generated representation should intervene. With the backbone, memory, generators, and readers frozen, a lightweight causal routing head is trained from counterfactual future-token likelihood advantages of GE and GH relative to E. At inference time, an admitted candidate modifies the E residual through bounded interpolation, while rejection recovers the direct pathway exactly. On question answering, MemoryAthena raises the five-task average from 37.65 to 39.28 over the direct pathway of the same checkpoint, while the six-task general-NLP average increases from 76.73 to 79.13. The complete memory-side system contains approximately 201M parameters, excluding the frozen backbone. Further analyses show complementary strengths among E, GE, and GH across tasks and inputs. These results support generated memory as a selective correction to direct retrieval and highlight routing when, which, and how strongly to intervene as the central challenge.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.25853