Common-Mode Collapse and Recovery in Direct Feedback Alignment
Direct feedback alignment can stall when a shared error mode saturates tanh hidden units.
Direct feedback alignment trains hidden layers with fixed random projections of output error. With tanh units and independent sigmoid outputs, plain SGD can stall near a constant class-frequency predictor because the error's common mode drives units toward saturation. A mean-covariance decomposition and a parameter-free reduced model predict activation-sensitivity concentration across 48 settings. On MNIST, class decodability largely survives, but learning stays slow unless the readout is calibrated to the class prior or the batch-mean error is removed; related effects occur on CIFAR-10.
- Shared error forms a rank-one update that saturates tanh units.
- Random feedback does not systematically correct that common mode at initialization.
- Class-prior readout calibration and batch-mean subtraction suppress collapse.
- Adam learns faster than fixed-rate SGD despite deeper collapse on MNIST.
- Similar effects appear in deeper and convolutional nets and on CIFAR-10.
From the original article
1 min read at arxiv.orgDirect feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs.
By Varun Reddy · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.