PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations
PrivDrift finds user secrets remain recoverable in LLM chats after topic drift, with 38.7–54.6% leakage.
PrivDrift is a benchmark for whether secrets a user discloses to an LLM remain recoverable after the conversation shifts topics and after persuasion-style probing. It contains 1,000 controlled multi-turn dialogues with seeded secrets, content-dense drift turns, and standardized extraction probes. Across three unnamed LLMs with extended context windows, dialogue-level hybrid leakage ranged from 38.7% to 54.6% and varied by model, secret type, and persuasion intensity. Within the tested window, more topic drift did not reliably reduce leakage.
- Benchmark uses 1,000 multi-turn dialogues with seeded secrets and drift turns.
- Hybrid leakage ranged from 38.7% to 54.6% across three long-context LLMs.
- Additional topic drift did not reliably reduce secret recoverability.
From the original article
1 min read at arxiv.orgLarge language models increasingly operate as persistent assistants in user-facing, shared-session, and tool-augmented settings. When users disclose sensitive information during an active conversation, that information may remain behaviorally recoverable through later prompts even after the dialogue shifts to unrelated topics.
By Luciano Maldonado · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.