Two-ReLU gradient descent collapses to bisector then specializes
Gradient flow on two-neuron ReLU model first collapses to bisector saddle before escaping to specialize to teacher.
Stories per day · 30 days
2 RSS
Relatedhopfield-networks1multi-index1multiple-instance-learning1neural-network-theory1phase-transitions1relu1softmax1
8 stories, newest first · one card per story
Stories tagged #gradient-descent
Paper maps phase transitions in max-pooled multiple instance learning
A minimal model shows freezing and retrieval transitions as softmax sharpness rises, and simulations match the predicted flow.
Differential Privacy of Gradient Descent on Perturbed Objectives
Proves explicit privacy bounds for finite gradient-descent iterates on perturbed objectives, removing ambient-dimension factors for generalized linear models.
QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency
QAM merges checkpoints to match sequential gradient descent through second order, with an O(h^3) error bound.
Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization
Priority-Constrained Descent optimizes hierarchical objectives while limiting distortion of the primary gradient.
AdamX: Cosine similarity meets gradient descent
Researchers propose AdamX, a cosine-similarity-based first-order optimizer with variance rectification that matches Adam-class convergence across benchmark datasets and architectures.
Silver Rate Is (Almost) Optimal for Gradient Descent Acceleration
New optimization theory paper proves near-optimal lower bounds for gradient descent with predetermined stepsizes, confirming silver-schedule optimality.
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Theoretical framework bounds generalization for gradient descent with weight decay, deriving conditions that explain delayed generalization and grokking.