HEAMS benchmarks agent memory for privacy, poisoning and user control
Original title: HEAMS: Holistic Evaluation of Agent Memory Systems
No tested system leads every performance benchmark, and strong task scores can coexist with weak privacy and security.
The paper presents HEAMS, an open-source framework that treats agent memory as the custodian of persistent user information. It states desiderata spanning performance, efficiency, security, privacy and user sovereignty, assembles a benchmark suite, and introduces the MilaMem prototype plus a governance wrapper for any memory system. Recall policies reduce inappropriate disclosure at a helpfulness cost, while recall and retention policies improve poisoning resistance.
Unconfirmed: Findings come from the authors' own benchmark comparisons, not an independent audit.
- Stored memory can be disclosed, resist deletion, outlast user intent, or be poisoned.
- Governance changes outcomes beyond raw task performance and efficiency.
- The authors say remaining gaps in the benchmark suite are acknowledged.
- Code is released at github.com/milatechtransfer/heams.
From the original article
1 min read at arxiv.orgMemory systems allow AI agents to persist information across interactions, making them more personalized and more capable on long-horizon tasks. This persistence introduces new risks: stored information can be disclosed in inappropriate contexts, resist correction or deletion, be retained beyond what users desire, or be poisoned to influence later agent behaviour.
By Chirag Goel · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.