Theory: noise can shape what linear RNNs learn
Original title: A theory of noisy linear recurrent neural network learning dynamics
In the rich regime, high noise can make a non-degenerate solution suboptimal when parameter scales are limited.
A theory paper analyzes noisy linear recurrent neural networks with white-noise inputs and argues that intrinsic noise can change representation learning, not merely add nuisance variation. In an analytically tractable model, noise can break solution-space degeneracies and, in some cases, determine whether a non-degenerate solution exists. In the rich regime, where weights start small, noise strongly changes how read-in, readout, and recurrent weights align. The authors also describe a phase transition: above sufficient noise, a non-degenerate solution is not optimal if parameter scales are limited, for example by regularization.
- The model uses a linear activation and white noise inputs.
- Noise modulates alignment of read-in, readout, and recurrent weights.
- Results extend recent analytic work on RNN representation learning.
- The analysis is theoretical rather than an empirical model release.
From the original article
1 min read at arxiv.orgBiological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning.
By John J. Vastola · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.