New LoRA Skills Should Read but Never Write
READ composes LoRA adapters so new skills can read old subspaces but cannot overwrite them.
The paper argues that composing independently trained LoRA adapters fails because equivalent factorizations and bidirectional coupling let new skills interfere with old ones. READ rewrites each adapter into a balanced canonical form that preserves its update exactly, then learns a one-directional coupling so a new skill can read old input subspaces without writing into their output subspaces. Only the new skill's coupling row is trained, and the composed update merges into the base weights with no extra inference cost or routing. Across four benchmark suites and two model families, READ beats baselines built from the same adapters by more than 20 points on SuperGLUE and more than 7 on a domain suite.
- Each adapter is rewritten into a balanced form that preserves its update.
- One-way coupling lets new skills read but not overwrite old skills.
- The composed update folds into base weights with no routing cost.
- Gains exceed 20 SuperGLUE points and 7 domain-suite points.
From the original article
1 min read at arxiv.orgLow-rank adapters (LoRA) make it cheap to fine-tune a large language model once per task, but combining several independently trained adapters into one model remains difficult: merging the updates in weight space causes interference, retraining on all task data is expensive, and routing between separate adapters gives up the goal of a single combined model.
By Zeyan Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.