Pseudonymisation tools leave personal values visible in most test documents
Original title: Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine
On a held-out set, the best existing tools left a whole value visible in 61% of Russian and 58% of UK English documents.
An open benchmark compares five pseudonymisation tools and a regular-expression baseline on synthetic Russian and UK English documents prepared for external LLMs. No existing tool meets all predefined thresholds on external datasets. An engine built from the two strongest tools, scored by its own author, leaves a whole value visible in 0.3% of Russian and 0.2% of UK English held-out documents. Real documents were not tested.
Unconfirmed: The benchmark was run by the engine's author, and real documents were not tested.
- The last held-out set contained 1,000 synthetic documents per language.
- In UK English the engine restores at least 99% of placeholders after 760 answers.
- Russian name-placeholder restoration is 94.6%, short of the 99% threshold.
- Wrong surname forms come from inflection borrowed from pii-guard.
- The engine was developed on six earlier held-out sets from the same generator.
From the original article
1 min read at arxiv.orgProfessionals send working documents to external large language models. A common safeguard is reversible pseudonymisation: personal data are replaced with placeholders before the request and restored locally in the answer. We present an open benchmark of five existing open tools and a regular-expression baseline on Russian and UK English, and an open engine built on the two strongest tools.
By Vladimir Beskorovainyi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.