PaLoRA: Paced Low-Rank Adaptation for Continual Learning
PaLoRA paces LoRA continual-learning steps by effective rank, reducing forgetting on long task sequences.
PaLoRA studies catastrophic forgetting in LoRA continual learning and argues that fixed small learning rates cannot contain leakage as the effective rank of past updates grows. Under an anisotropic leakage model, the authors derive a pacing law, s-star equals R over c, that tightens gradient-step magnitude as historical rank R increases. PaLoRA compresses prior knowledge with adaptive SVD truncation, projects gradients into the nullspace of earlier tasks, and applies rank-aware pacing. It reports roughly 4 percentage points higher accuracy than prior methods on 50-task ImageNet-A and ImageNet-R.
- Fixed small learning rates do not stop rank-driven forgetting.
- Derived pacing law sets step scale to effective rank over a constant.
- Method uses SVD truncation, nullspace projection, and adaptive pacing.
- Reports about 4% gains on 50-task ImageNet-A and ImageNet-R.
From the original article
1 min read at huggingface.co
LoRA-based continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate.
By Yuxuan Li, Fanhu Zeng, Hao Tang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.