Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks
Net Utility allocates LoRA merge rank by utility and interference, beating uniform budgets on vision and language.
The paper argues that giving every layer and task the same LoRA rank budget causes much of the gap between merged adapters and per-task LoRAs. Net Utility decomposes each task adapter with SVD and scores singular directions by task utility and interference with other tasks, then selects the highest-scoring directions under a total budget without using data. Applied on top of five merging methods and three merging spaces for vision and language task sets, it improved average performance by 2.1% on vision and 2.2% on language.
- Uniform layer and task rank budgets are identified as a major merging performance gap.
- Net Utility scores SVD directions by task utility and interference without training data.
- Highest-scoring directions are selected under a global rank budget.
- Average gains were 2.1% on vision tasks and 2.2% on language tasks.
- The metric was applied to five merging methods across three merging spaces.
From the original article
1 min read at huggingface.co
Merging low-rank adapters (LoRAs) promises to eliminate the overhead of swapping task-specific weights at inference time. However, existing merging methods assume every layer needs the same rank budget. Further, some methods assume that rank budget needs to be split equally among the tasks too.
By Avinash Amballa, Yashas Malur Saidutta, Wenbo Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.