LoRA merge audit finds bad adapters inflate forgetting scores
Naive merging of an audited four-adapter set cut GSM8K by 10.5 points and HumanEval by 18.3.
Stories per day · 30 days
2 RSS
Relatedlora2ai-experts1asr1catastrophic-forgetting1continual-learning1fine-tuning1hypernetwork1llama1mistral1multilingual1nestedvla1robotics1skill-consolidation1speech-llm1vla1
13 stories, newest first · one card per story
Stories tagged #model-merging
Model merging succeeds with non-uniform LoRA ranks
Domain-specific LoRA ranks improve multi-domain accuracy without coupling or selection conflicts in merging methods.
Speech LLM ASR expansion forgets source languages unless merges are selective
SSU-Wanda+TIES best preserved source languages in single-target expansion; recursive multi-target updates were highly order-sensitive.
NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models
NestedVLA generates task-specific VLA parameters, beating model merging by 41 points across three robotics benchmarks.
To Align or Not To Align: Check Your COMPASS Before You Train
COMPASS predicts model-pair compatibility before alignment or merging, correlating 0.91 with cross-modal success.
QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency
QAM merges checkpoints to match sequential gradient descent through second order, with an O(h^3) error bound.
SMAT: Simple and Efficient Merge-Aware Training
SMAT raises merged-model scores by 1.07–2.16 points with under 2% extra training cost.
Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
Researchers find MoE routing drift after merging does not reliably indicate routing failure.
Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
Routing drift after MoE merging does not reliably indicate failure; source-route restoration often lacks task benefit.
New LoRA Skills Should Read but Never Write
READ composes LoRA adapters so new skills can read old subspaces but cannot overwrite them.
ACLArena: Agent Continue Learning in Multi-stage Post-training
ACLArena compares agent continual-learning methods and proposes replay plus routed LoRA experts to limit forgetting.
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
Climate-ModernBERT domain-adapted encoders reach 76.3 average F1 across nine climate benchmarks, 2.8 points above vanilla ModernBERT-Base.
Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks
Net Utility allocates LoRA merge rank by utility and interference, beating uniform budgets on vision and language.