Spectral unlearning applies matrix-structured updates to transformers
Original title: Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models
Applied atop existing objectives, it narrows most of the gap to a retain-trained reference on memorization, privacy, and utility.
The paper proposes spectral unlearning, combining standard unlearning objectives with matrix-structured optimization of transformer weights. The authors say the spectral update is the steepest-descent direction for linear layers with normalized inputs and outputs. Evaluations use a harmonic mean of memorization, privacy, and utility, and code is published on GitHub.
Unconfirmed: Outperformance is the authors' comparison against their chosen baselines.
- The method is applied on top of existing unlearning objectives.
- Results are summarized with a harmonic mean of memorization, privacy, and utility.
- Code is available at github.com/unlearn-ml/spectral-unlearning.
From the original article
1 min read at arxiv.orgMachine unlearning in large language models (LLMs) aims to remove specific learned knowledge, capabilities, or behaviors while preserving model utility. Existing methods have mostly focused on the design of unlearning objectives, while largely overlooking the intrinsic matrix structure of transformer weights.
By Sung Il Choi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.