How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape
A theory paper shows frozen-activation bottleneck rank, not sample size, controls why LoRA updates stay low-rank.
The paper gives a non-linearized explanation for why low-rank adaptation works: once the backbone is frozen, the loss observes each adapted weight only through a finite-rank map induced by frozen activations, called the bottleneck rank. Trace-norm regularization then selects a low-rank matrix among matrices with the same observed output, with tighter bounds for multi-head attention from softmax row invariance and frozen output projection. For factorized LoRA with weight decay, a small generic perturbation makes every first-order stationary point rank-deficient once LoRA rank exceeds a bottleneck-rank threshold, and the authors provide near-optimality and uniform generalization bounds.
- With a frozen backbone, each adapted matrix is seen through a finite-rank map.
- Trace-norm regularization selects low-rank solutions among equivalent outputs.
- Attention softmax and output projection yield tighter rank bounds.
- Above a bottleneck-rank threshold, stationary LoRA points become rank-deficient.
- Generalization bounds depend on the spectral structure of frozen features.
From the original article
1 min read at arxiv.orgLow-rank adaptation (LoRA) has become the standard method for parameter-efficient fine-tuning of large pretrained models, yet theoretical explanations for why low-rank updates suffice remain incomplete. Most existing analyses rely on the Neural Tangent Kernel (NTK) regime, which linearizes the network around the pretrained weights.
By Long Nguyen-Chi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.