LEAN regularizer reshapes weight sensitivity to cut bit-flip damage
Original title: LEAN: Loss-Engineered Modulation of the Model Weight-Sensitivity Distribution for Efficient, Error-Tolerant Neural Networks
Ten epochs shrink the sensitive tail 5 to 13 times on nine vision models while staying within 0.5 accuracy points.
LEAN adds a composite regularizer on gradient and storage-format flip step so a trained network's weight-sensitivity distribution can be shifted. Real bit flips on the reshaped weights cause less damage, and a plain gradient-norm penalty does not reproduce the effect. With a BCH code over 1% of weights, models survive 300 progressive bit-search flips in every training seed, while three of four published defenses with the same code fail by 300. The approach also cuts parity overhead 3.7 times for a fixed guarantee.
Unconfirmed: Language-model results are described as a preliminary extension.
- The most sensitive 1% of weights carry about a fifth of bit-flip damage.
- Tests cover ResNet-18, ResNet-50 and VGG-16 on CIFAR-10, CIFAR-100 and ImageNet.
- Models survive 500 flips in two of three seeds; WCAT falls to 75% at 500.
- ImageNet ResNet-50 stores at a 3.29-bit payload, 2.4 times smaller than INT8.
- A preliminary extension to three small language models shows the same tail shaping.
From the original article
1 min read at arxiv.orgThe weights of a trained network do not have uniform sensitivity to model accuracy: the 1% most sensitive weights carry about a fifth of the damage a bit flip can do. This distribution decides the bit-flip tolerance of a model, the fraction of weights a code must cover and the bits each weight needs; existing schemes work with the distribution training leaves.
By Habibur Rahaman · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.