Local Support Learning
Local Support Learning limits catastrophic forgetting in LLMs up to 7B without replaying prior data.
Local Support Learning treats catastrophic forgetting as a geometric problem in each weight matrix's input space and adds a retention mechanism that does not require prior data. During new training, a standard weight adapter minimizes the loss while a Gaussian-mixture gate enables that adapter only on activations from the current distribution. The authors show the post-training method can preserve both pretrained and finetuned capabilities across multiple phases in LLMs up to 7 billion parameters, with low memory and compute cost and robustness to hyperparameters.
- A standard adapter minimizes loss while a GMM gate keeps the update local.
- The gate trains only on current data yet stays closed on earlier phases.
- The method retains pretrained and finetuned skills in models up to 7B.
- It needs no prior data and is efficient and robust to hyperparameters.
From the original article
1 min read at huggingface.co
We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal.
By Assaf Ben-Kish, Akarsh Kumar, James Glass · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.