BISECT improves low-rank continual learning for Vision Transformers
Original title: BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning
BISECT partitions rank budget into invariant carrier and novelty complement, outperforming baselines with no extra parameters.
Prevailing LoRA methods cause spectral starvation in continual learning due to strict subspace isolation. BISECT uses bilevel spectral decomposition with invariant carrier and novelty complement, achieving better performance on ImageNet-R, ImageNet-A, CIFAR-100, CUB-200, DomainNet without extra inference cost.
- historical null tails contain barely 5.5% of incoming task variance
- BISECT outperforms prompt, adapter, and LoRA baselines
- zero extra inference parameters or FLOPs
- remains complementary to sparse analytic expansion
From the original article
1 min read at arxiv.orgExemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space through strict subspace isolation. By projecting each incoming task into the approximate null space of historical token activations, isolation-based learners equate any cross-task feature overlap with catastrophic interference.
By Tom Saliencro · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.