LoRA dynamics differ between supervised fine-tuning and reinforcement learning
Original title: Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning
RL concentrates LoRA spectrum while SFT expands it at moderate rates, with group-mean baseline subtraction identified as the source.
Tracking the singular spectrum and subspaces of LoRA updates on mathematical reasoning tasks reveals RL concentrates the spectrum at every learning rate and rotates singular subspaces faster than SFT at small learning rates. SFT expands the spectrum at moderate rates. A quadratic multi-index model updated via LoRA explains the mechanism, guiding tailored LoRA configurations for each stage that yield best performance in most settings.
- RL concentrates spectrum across learning rates while SFT expands at moderate rates
- RL rotates singular subspaces much faster than SFT at small learning rates
- Group-mean baseline subtraction tilts balance toward spectral concentration
- Custom LoRA updates for each stage outperform compared methods
- Tested on mathematical reasoning tasks across multiple models
From the original article
1 min read at arxiv.orgLow-rank adaptation (LoRA) is commonly used in both supervised fine-tuning (SFT) and reinforcement learning (RL), often under similar optimizer settings. Yet, how the different dynamics of the two stages manifest in the low-rank increment that LoRA optimizes remains unclear.
By Chenruo Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.