Continual Learning Mechanisms Compose for Long-Horizon Memorization
Composing data, function, and weight anchors with merged LoRA raises 100-task long-horizon retention from 1.2% to 34.9% in continual fine-tuning.
The paper introduces long-horizon memorization: a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier examples or receiving task identifiers at inference. No single continual learning mechanism maintains strong retention at this horizon, so the authors compose complementary mechanisms along data/function/weight anchors and low-rank allocation rules. The best method combining all three anchors with merged LoRA ranks among the top 3 methods on all three datasets and raises average final retention from 1.2% to 34.9%, a 28-fold improvement.
- 100-task continual fine-tuning setting without task IDs or replay
- Composes data, function, and weight anchors with low-rank allocation rules
- Best method lifts average retention from 1.2% to 34.9%, a 28-fold gain
- Data anchor and merged LoRA interact super-additively across all three datasets
Full article221 words · extracted from huggingface.co · click to collapse
Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-level successive halving to search the combinatorial design space and use a factorial experiment to measure individual and interaction effects. Our best method combines all three anchors with merged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.06986