When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Spurious forgetting hides old facts through a reversible shared shift before slower fact-specific erosion.
The paper analyzes spurious forgetting, where facts a language model seems to lose during finetuning remain stored and can be recovered. In one regime, recall of old facts collapses, returns as training continues on new facts, and only later erodes permanently. A minimal associative memory reproduces this with shared key structure, concentrated new values, and normalization: finetuning shifts old representations together, and normalization later withdraws that shift. Subtracting the common shift removed the collapse in a synthetic Transformer, and removing one direction from each weight update restored old facts in a pretrained language model.
- Knowledge that appears forgotten during finetuning can often be recovered.
- Recall can collapse, recover during continued training, then erode permanently.
- A shared representation shift hides facts while preserving their relative geometry.
- Removing one update direction restored old facts in a pretrained language model.
From the original article
1 min read at arxiv.orgKnowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic.
By Vedant Palit · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.