Model merging succeeds with non-uniform LoRA ranks
Original title: Model Merging Does Not Require Uniform LoRA Ranks
Domain-specific LoRA ranks improve multi-domain accuracy without coupling or selection conflicts in merging methods.
Model merging combines separately fine-tuned experts into a single model. Standard practice trains every expert at the same LoRA rank, coupling choices across domains. The study trains experts at four ranks on three domains then merges with six methods. Non-uniform ranks yield up to 1.8 percentage points higher multi-domain accuracy while keeping methods effective.
- Non-uniform ranks cost nothing in averaging and selection
- Magnitude-based component choice discards useful adapters
- Loss-based selection maintains flat accuracy across rank range
From the original article
1 min read at arxiv.orgModel merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging.
By Zirong Xie · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.