Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss
A local learning interaction unifies data attribution, forgetting, and plasticity loss in continual updates.
The paper derives a token- and layer-wise decomposition of how learning from one token changes another prediction, separating softmax force, shared readout geometry, and residual connections into two interaction channels with a forward-computable approximation. Following that interaction over time unifies data attribution, forgetting, and plasticity loss: positive interaction identifies useful experience, negative interaction produces collision or erosion, and later updates reshape the geometry that carries future learning signals. The same local mechanism supports data selection, interference controls, and a readout-based diagnostic whose degradation predicts the benefit of restoring the readout.
- Token- and layer-wise decomposition of how one update changes another prediction
- Separates softmax force, shared readout geometry, and residual connections
- Positive interaction marks useful data; negative interaction causes forgetting
- Longer-horizon updates reshape geometry and reduce future learning signals
- A readout diagnostic predicts when restoring the readout helps
From the original article
1 min read at huggingface.co
Modern language models are likely to be updated throughout their lifetime rather than trained once and frozen. Each update therefore participates in a recurring cycle: decide which experience to learn from, understand what that update changes, and remain capable of learning from what comes next. We show that these challenges are governed by the same evolving update--behavior interaction.
By Yi Ren, Wenlong Deng, Guanzhe Hong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.