MemJoin benchmarks cross-session privacy leaks in agent memory
Original title: Can Agents Keep Secrets They Were Never Told? Benchmarking Latent Privacy Risks Across Sessions with MemJoin
In 288 synthetic cases, a complete history reveals a sensitive relation that no single session contains.
The paper introduces MemJoin, a benchmark of agent memory systems that retain user information across sessions. Each case is built so combining sessions entails a sensitive person-linked relation, while every individual session is insufficient. Six memory systems were evaluated with one answer-generating model, plus full-history and no-history controls. A paired test found that removing connecting clues can change whether the complete relation is stated even when checked attribute values remain visible.
- The benchmark contains 288 synthetic analysis cases.
- Target annotations stay with the evaluator and are not given to the systems.
- Authors separately check stored records, retrieved context, and the final answer.
- Attribute visibility alone missed what records from different sessions reveal together.
From the original article
1 min read at arxiv.orgTo support sustained interactions and personalized assistance, agents use long-term memory to retain and combine user information across sessions. Reusing such memories can lead an agent to state a sensitive relation while completing an ordinary task that does not require it.
By Qishun Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.