Broken scale symmetries in undercomplete linear autoencoders
Theory paper shows SGD drives undercomplete linear autoencoders toward large decoder weights along the PCA solution manifold until hitting a finite-stepsize stability boundary.
Stories per day · 30 days
7 stories, newest first · one card per story
Stories tagged #deep-learning-theory
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Paper derives a spectral theory showing weight decay drives NTK feature learning that controls grokking timescale and phase structure.
Sharp Reconstruction Bounds for Autoencoders Using the Same Forward Map
Theoretical work derives sharp reconstruction-derivative error bounds for autoencoders using the same forward map, validated on a 798,452-point LiDAR forest scan.
Special Lagrangian cones in Deep Learning
A mathematics paper generalizes Harvey-Lawson cones, proving a matrix version is an exact special Lagrangian manifold foliating the balanced manifold in deep learning.
Benign Loss Landscapes Can Coexist with Worst-Case Hardness
Theory paper shows tree tensor networks contain worst-case hard targets yet benign loss landscapes, with difficulty arising from degenerate saddles.
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Researchers derive an exact law linking learning-rate schedules and weight decay in normalized networks, pinpointing when scale-invariant optimization destabilizes.
Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics
Researchers model curriculum learning as Wasserstein transport over difficulty distributions, finding curriculum benefits are strongly task- and budget-dependent with no dominant strategy.