Inner Momentum for Differentially Private Muon
Averaging per-example Muon gradients over recent model snapshots before clipping bounds distortion and improves differentially private GPT-2 fine-tuning.
25
Averaging per-example Muon gradients over recent model snapshots before clipping bounds distortion and improves differentially private GPT-2 fine-tuning.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views