"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverables
LLM revision comments can leak withdrawn secrets, recoverable from about 13% of tested deliverables.
The paper defines revision traces, where an LLM removes sensitive content from a deliverable but restates the edit so a recipient can recover it. In three public conversation corpora, 2,363 of 26,753 revision requests (8.8%) left traces. On RevLeakBench, 100 tasks with conversation and agent tracks, about half of deliverables from six models stated the edit, and a reader seeing only the deliverable recovered the withdrawn item from about 13%; warning that the full reply will be forwarded still left traces in 36.4%. An output-side filter sharply reduces recovery with little loss of required content.
- 8.8% of 26,753 revision requests left revision traces.
- RevLeakBench has 100 tasks across conversation and agent tracks.
- Readers recovered withdrawn items from about 13% of deliverables.
- A forwarding warning still left traces in 36.4% of outputs.
- An output-side filter cuts recovery with little content loss.
- Products
- RevLeakBench
From the original article
1 min read at arxiv.orgLarge language model (LLM) assistants increasingly help users draft content for third-party recipients. During private drafting, the user or the model may introduce an item and later remove or replace it. The model may remove the item from the intended content but reveal it again when stating the edit.
By Yage Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.