Prediction Is Not a Sufficient Memory State
Theory shows a fixed recurrent memory matching current predictions can still be insufficient for future learning.
The paper argues that a memory reproducing every current prediction may still lack the state required to learn from the next observation. For smooth convex residual losses, an exact fixed-dimension continuous recurrent state exists only when the loss is polynomial, with degree D requiring r = binom(d+D, D) − 1 coordinates. Ridge regression's exact state costs dp + d(d+1)/2 versus dp for the predictor, and kernel attention receives a second exact-closure classification. The results characterize future updates rather than proposing a new algorithm.
- Matching the current predictor can still omit state needed to keep learning
- Fixed-dimension exact recurrent memory exists only for polynomial losses
- Degree D needs binom(d+D, D) minus 1 coordinates
- Ridge regression's exact state also stores the design matrix
From the original article
1 min read at arxiv.orgA memory that reproduces every current prediction may still have forgotten how to learn from the next observation. We characterize this distinction for exact cumulative learning and attention. For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial.
By Robert Jenkinson Alvarez · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.