AfterDeleteBench finds LLM agents can recover deleted secrets
Original title: AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents
One query recovered a deleted secret from Qwen2.5-7B-Instruct in about two thirds of cases, while never-exposed controls recovered none.
Researchers introduce AfterDeleteBench, a controlled audit of whether fixed-weight LLM agents still carry a secret after the original memory record is deleted. Each trial compares an exposed-then-deleted agent with a matched agent that was never exposed. Exact reconstruction succeeded in 62–99% of cases across Qwen, Mistral, and Phi-3 when only separately stored components remained under a public composition rule. Provenance-aware closure removed observed recovery in the tested positive-baseline settings, while an exact-literal write filter only blocked reinsertion.
Unconfirmed: Findings are from a controlled synthetic-secret audit, not a reported production incident.
- Recovery does not require a surviving verbatim copy of the secret string.
- Misaligned reply formats and parsers can score target-bearing answers as failures.
- The study separates reconstruction from write-back and fresh-session recall.
- Deletion should be evaluated for the agent and its memory system, not storage alone.
- Products
- AfterDeleteBench
- AI models
- Qwen2.5-7B-InstructMistralPhi-3
From the original article
1 min read at arxiv.orgMany LLM agents maintain external persistent memory and derive additional state from past interactions, such as summaries. Deleting an original memory record therefore need not remove information carried by its descendants. We introduce AfterDeleteBench, a controlled behavioral audit for this failure mode in fixed-weight LLM agents.
By Ehsan Saleh · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.