QuasiLeak-Bench: Evaluating LLM-Based Privacy Detectors on Policy-Conditioned Quasi-Evidence Group Recovery
QuasiLeak-Bench shows a fine-tuned 4B model beats prompted LLM privacy detectors on quasi-evidence groups.
QuasiLeak-Bench measures whether privacy detectors recover complete quasi-evidence groups—weak quasi-identifiers that jointly reveal a private attribute—and flag disclosures banned by a sender-specified policy. Ground truth is produced by composing verified quasi-identifiers into documents, then applying automated quality control. Existing detectors transfer poorly and locally deployable LLMs miss most groups, but a 4B model fine-tuned on the training split surpasses the strongest prompted model on held-out documents.
- Quasi-evidence groups are weak cues that jointly reveal private attributes.
- Benchmark requires complete group recovery under sender disclosure policies.
- Existing detectors transfer poorly; local LLMs miss most groups.
- A fine-tuned 4B model beat the strongest prompted model.
From the original article
1 min read at arxiv.orgNatural language messages often contain quasi-identifiers that are weak in isolation but can jointly reveal an individual's private attributes. We call such a jointly revealing set of cues a quasi-evidence group. Existing privacy benchmarks focus on explicit identifiers, attribute inference, or task-driven disclosures, leaving inference from quasi-evidence groups largely unmeasured.
By Yufan Zhu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.