SMAT: Simple and Efficient Merge-Aware Training
SMAT raises merged-model scores by 1.07–2.16 points with under 2% extra training cost.
SMAT is a merge-aware training method that treats common model merging as scale, mask, and perturb operations from each expert's perspective. It jointly optimizes task loss and expected loss at simulated merged parameters sampled with scaling coefficients, masks, and additive noise. Periodic scheduling, kernel fusion, and parameter storage switching keep the method to one forward and one backward pass per step. Across four language and vision-language backbones, mean scores over five merging methods improve by 1.07–2.16 points over the strongest baseline, with under 2% training-time overhead versus standard fine-tuning.
- SMAT models merge effects as Scale, Mask, and Perturb operations.
- It optimizes expert loss plus expected loss at simulated merged parameters.
- Mean scores rise 1.07–2.16 points across five merging methods and four backbones.
- Overhead stays under 2% versus standard fine-tuning via fused, scheduled updates.
From the original article
1 min read at huggingface.co
Model merging integrates the capabilities of multiple experts without joint retraining, but standard expert training optimizes task loss alone and does not guarantee good performance after merging. Merge-aware training (MAT) aims to improve merged performance, but existing methods do not fully account for common merging operations and add training cost.
By Yanggan Gu, Yuanyi Wang, Zhen Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.