Study measures how neural representations form during training
Path tortuosity, not endpoint displacement, predicted later drift and sensitivity to parameter perturbations.
Stories per day · 30 days
9 stories, newest first · one card per story
Stories tagged #training-dynamics
Prospective Prediction of OOD Degradation from Source-Side Training Dynamics
Source-side training dynamics can prospectively predict later out-of-distribution model failure.
The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Weight decay causally regulates massive-activation scale in transformers, with peak magnitude scaling near lambda to the -1/2.
Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reduction
A monograph derives exact training and inference identities for power-law decoder representation language models.
Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss
A local learning interaction unifies data attribution, forgetting, and plasticity loss in continual updates.
Common-Mode Collapse and Recovery in Direct Feedback Alignment
Direct feedback alignment can stall when a shared error mode saturates tanh hidden units.
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Researchers derive an exact law linking learning-rate schedules and weight decay in normalized networks, pinpointing when scale-invariant optimization destabilizes.
Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics
Researchers model curriculum learning as Wasserstein transport over difficulty distributions, finding curriculum benefits are strongly task- and budget-dependent with no dominant strategy.
Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physics
Latent-MoE adds domain-aware mixture-of-experts routing to PINNs, improving accuracy over an order of magnitude on multi-regime physics PDEs.