Deletion Tuning matches full fine-tuning by zeroing under 0.05% of weights
Original title: The Surprising Effectiveness of Deleting Weights in LLM Post-Training
A binary mask on gate projections stays near base prior-task accuracy while FFT, LoRA, and random sparse tuning lose up to 5.4 points.
A paper reports that zeroing fewer than 0.05% of a pretrained LLM's weights, with no other modification, matches full fine-tuning across reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Deletion Tuning learns a binary mask over the gate projection of transformer blocks. The mask is hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoint and forgets less. Geometry analysis finds it closest to full fine-tuning versus LoRA and random sparse fine-tuning.
- The method changes only a learned binary mask, not other weights.
- Under RL, prior-task accuracy stays near the base level.
- FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points.
- Analysis uses stable rank, spectrum drift, and principal-weight overlap.
From the original article
1 min read at huggingface.co
How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0. 05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation.
By Jack Lu, Zhenbang Yang, Mike Lasby · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.