MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents
MemPilot uses an RL policy to curate multimodal agent memory on demand, trading off accuracy, cost, and latency.
MemPilot uses a multi-step LLM policy trained with reinforcement learning to decide between retrieving query-agnostic memory and curating raw multimodal history with heterogeneous LLMs and VLMs. The policy controls evidence amount, curation instructions, model selection, and visual access under explicit performance, cost, and latency preferences. Training uses objective-wise advantage decoupling and prefix-based marginal utility estimation for multi-step credit assignment. On five multimodal agent-memory benchmarks, preference sweeps produce broader trade-off frontiers than existing trade-off-aware baselines.
- RL policy chooses retrieval versus on-demand multimodal curation
- Controls evidence volume, instructions, model choice, and visual access
- Separately estimates advantages for performance, cost, and latency
- Broader trade-off frontiers than baselines on five benchmarks
From the original article
1 min read at arxiv.orgMemory has become integral to the LLM agent ecosystem, supporting information retention and reuse across interactions. However, most existing agent memory systems construct memory in a query-agnostic manner, which can incur unnecessary preprocessing cost and discard details that later prove essential.
By Haozhen Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.