Sakana AI Researchers Introduce PC-ALM, a Layer-Local Alternative to Backpropagation That Trains 1000-Layer Networks
Sakana AI's PC-ALM adds per-layer Lagrange multipliers to predictive coding, matching backprop on networks up to 1000 layers with layer-local updates.
Sakana AI researchers propose Augmented Lagrangian Predictive Coding (PC-ALM), a training method that keeps every update layer-local while recovering backprop-aligned credit signals. The team proves multipliers converge to exact backprop adjoints in linear networks and trains 1000-layer residual MLPs on MNIST within about 2 points of backprop accuracy. PC-ALM matched backprop across a width/depth grid from 8 to 128 on MNIST and Fashion-MNIST where standard predictive coding failed in deep, narrow networks, and improved over PC on ResNet-18 with CIFAR-10 and Tiny ImageNet. An MIT-licensed JAX reference implementation reproduces the results on CPU.