Sakana AI Researchers Introduce PC-ALM, a Layer-Local Alternative to Backpropagation That Trains 1000-Layer Networks
Sakana AI's PC-ALM adds per-layer Lagrange multipliers to predictive coding, matching backprop on networks up to 1000 layers with layer-local updates.
Sakana AI researchers propose Augmented Lagrangian Predictive Coding (PC-ALM), a training method that keeps every update layer-local while recovering backprop-aligned credit signals. The team proves multipliers converge to exact backprop adjoints in linear networks and trains 1000-layer residual MLPs on MNIST within about 2 points of backprop accuracy. PC-ALM matched backprop across a width/depth grid from 8 to 128 on MNIST and Fashion-MNIST where standard predictive coding failed in deep, narrow networks, and improved over PC on ResNet-18 with CIFAR-10 and Tiny ImageNet. An MIT-licensed JAX reference implementation reproduces the results on CPU.
- Layer-local training method avoiding backprop's global synchronization.
- In linear networks, Lagrange multipliers provably converge to exact backprop gradients.
- Trains 1000-layer residual MLPs within ~2 points of backprop on MNIST.
- MIT-licensed JAX code; tested only on small image benchmarks so far.
From the original article
3 min read at marktechpost.com
Sakana AI researchers Jeffrey Seely and Julian Gould introduce Augmented Lagrangian Predictive Coding (PC-ALM), a local-learning alternative to backpropagation. By attaching a Lagrange multiplier to each layer constraint, PC-ALM keeps predictive coding's layer-local updates while recovering exact backprop gradients in linear networks.
By Asif Razzaq · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.