LoRA merge audit finds bad adapters inflate forgetting scores
Original title: Audit Before You Merge: Provenance, Probing, and Continual LoRA Composition
Naive merging of an audited four-adapter set cut GSM8K by 10.5 points and HumanEval by 18.3.
The paper studies catastrophic forgetting when public LoRA adapters are merged onto Llama-3.1-8B-Instruct. It proposes checks for base-model match and a non-trivial delta norm, plus leakage-free probing, before measuring forgetting. Near-zero-norm and mismatched adapters inflated interference by about 10 percentage points. Task Arithmetic at lambda 0.5 stayed within a standard error of the unmerged baseline, and the ranking replicated on Mistral-7B-Instruct-v0.3.
Unconfirmed: The roughly 10 percentage-point interference inflation is the authors' measurement.
- Public hubs included silent base-version mismatches and near-zero delta norms.
- A coordinate search marked the math adapter probe-optimal at lambda -0.5.
- Two of four Mistral candidate adapters failed the audit.
- The audited Mistral merge reproduced the same method ranking on GSM8K.
From the original article
1 min read at arxiv.orgPost-hoc merging of public LoRA adapters is an attractive gradient-free path to continual adaptation, but reliable measurement of catastrophic forgetting in this setting depends on infrastructure that prior work has not addressed. We treat each adapter merge as a sequential task in a continual-learning setting, with the instruction-tuned base model's capabilities as Task 0, and study forgetting on Llama-3.
By Lenny Aharon · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.