Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Researchers introduce FOM-UL, a layer-selective machine unlearning framework that improves forgetting-utility trade-offs and resists knowledge recovery after quantization.
FOM-UL selects transformer layers for unlearning using a forget-to-retain significance score, concentrating parameter updates on layers highly influential for the forget set while leaving most of the model unchanged. It reduces residual memorization versus GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines on TOFU, KnowUnDo, and MUSE-style evaluations while preserving retain-set utility. Under 8-bit and 4-bit post-training quantization and adversarial prompts, it maintains stronger suppression of forgotten content, addressing brittleness of diffuse unlearning updates.
- Layer-level unlearning via forget-to-retain significance scoring
- Improves forgetting-utility trade-off over existing unlearning baselines
- Maintains suppression after 8-bit and 4-bit quantization
- Adversarial prompt evaluations show lower recovery of forgotten content
- Evaluated on TOFU, KnowUnDo, and MUSE benchmarks
From the original article
1 min read at arxiv.orgLarge Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns.
By Ravi Ranjan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.