Continual Learning Mechanisms Compose for Long-Horizon Memorization
Composing data, function, and weight anchors with merged LoRA raises 100-task long-horizon retention from 1.2% to 34.9% in continual fine-tuning.
The paper introduces long-horizon memorization: a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier examples or receiving task identifiers at inference. No single continual learning mechanism maintains strong retention at this horizon, so the authors compose complementary mechanisms along data/function/weight anchors and low-rank allocation rules. The best method combining all three anchors with merged LoRA ranks among the top 3 methods on all three datasets and raises average final retention from 1.2% to 34.9%, a 28-fold improvement.
- 100-task continual fine-tuning setting without task IDs or replay
- Composes data, function, and weight anchors with low-rank allocation rules
- Best method lifts average retention from 1.2% to 34.9%, a 28-fold gain
- Data anchor and merged LoRA interact super-additively across all three datasets
From the original article
1 min read at huggingface.co
Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference.
By Zheyuan Zhang, Alvin Zhang, Daniel Khashabi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.