Two-ReLU gradient descent collapses to bisector then specializes
Gradient flow on two-neuron ReLU model first collapses to bisector saddle before escaping to specialize to teacher.
Stories per day · 30 days
Relatedcifar-101deep-learning1expressivity1gaussian1gradient-descent1hermite-tensors1information-theory1learning-theory1model-distillation1model-extraction1multi-index1mutual-information1neural-networks1neural-network-theory1pac-learning1parameter-recovery1piecewise-linear1representation-learning1
11 stories, newest first · one card per story
Stories tagged #relu
Paper estimates ReLU-net mutual information from routing over linear regions
The routing term uses a plug-in estimator and, when well sampled, approximates a lower bound on joint information.
Paper recovers neural-network parameters by extraction and distillation
On the largest synthetic net, Leaky ReLU recovered 51-57 deepest-layer neurons while ReLU recovered none.
Fully polynomial algorithm learns positive linear combinations of ReLUs
The method uses poly(d, k, 1/ε) samples and time under Gaussian inputs and L2 loss, without well-conditioned weight assumptions.
Any ReLU network can be exactly simulated with two hidden layers
The authors give a homological-algebra proof using fillings and a separate direct combinatorial proof.
Constructive Bounds for the Topological Expressivity of Neural Networks
A paper proves ReLU architectures can realize prescribed Betti numbers, with tighter bounds for products of spheres.
Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking
A symmetry-breaking account explains clustered Hessian spectra in trained deep networks, including ReLU, CNN, graph, and transformer models.
Security Properties of Neural Networks as Decision Problems
Researchers classify neural-network security checks as co-NP-complete or harder, including backdoors and parameter faults.
Beyond Scalar Probes: Exploiting Vector-Valued Outputs in ReLU Networks For Signature Extraction
Vector-valued Jacobian analysis improves cryptanalytic signature extraction from ReLU networks at low precision.
Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections
Researchers characterize minimal-norm univariate two-layer ReLU classifiers and show skip connections make every KKT point globally optimal.
End-to-End Hard-Label Cryptanalytic Model Extraction Using Efficient Sign Recovery
Researchers show end-to-end hard-label extraction of small ReLU MLPs, exceeding 98% label agreement on MNIST.