Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning
A 174-language benchmark shows LLM unlearning fails across languages, and COVER improves cross-lingual erasure.
The paper shows that unlearning a fact in one language does not reliably erase it in others: changing the query or the requested answer language can reopen the knowledge. It introduces language-budgeted multilingual unlearning and the Cross-Lingual Unlearning Tensor, spanning 174 language-script pairs and 25 paraphrase types. COVER selects source languages to maximize predicted coverage of languages that receive no forget supervision, using only benign calibration data and a frozen model at deployment. Across three model families and two forget sets, it cuts mean held-out residual access by 7.8–27.3% relative to uniform selection, with similar gains on human-translated LORELEI news.
- Changing query or answer language can restore supposedly forgotten facts.
- The benchmark covers 174 language-script pairs and 25 paraphrase types.
- COVER picks a language subset to maximize cross-lingual erasure.
- Held-out residual access falls 7.8–27.3% versus uniform selection.
- Gains also appear on human-translated LORELEI news in low-resource languages.
From the original article
1 min read at arxiv.orgUnlearning a fact in one language does not guarantee its removal in others as changing the query or even the requested answer language can reopen seemingly forgotten knowledge -- a cross-lingual loophole. The most straightforward solution to this challenge -- unlearning in all languages -- is neither scalable nor desirable as it amplifies damage to unrelated model capabilities.
By Tyler Skow · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.