Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks
Researchers derive functional-gradient rules for retrieval-centric layers that grow by storing key-value pairs for every training point.
The paper proposes retrieval-centric deep learning, in which layers store a key-value pair for every training example and recombine those representations with attention at inference, so the network grows with the data. It shows that naively transferring linear-attention learning rules to richer kernels is not principled optimization, then derives functional-gradient rules for RBF and softmax-like kernels. Experiments cover image classification and synthetic teacher-student tasks. Replacing linear attention with MesaNet or DeltaNet also links this dual view to recent optimizers for fixed-size networks.
- Layers store one key-value pair per training point
- Functional gradients derived for RBF and softmax-like kernels
- Tested on image classification and teacher-student tasks
- MesaNet and DeltaNet connect the dual view to fixed-size optimizers
From the original article
1 min read at huggingface.co
We investigate a general-purpose layer for deep learning that, instead of compressing arbitrary-size training data into fixed-size weight matrices, stores a new pair of key-value representations for every data point during training, and retrieves and recombines these representations through an attention mechanism at inference time - resulting in a growing neural net (NN). While Irie et al. (arXiv:2202.
By Maximilian Schlegel, Rajai Nasser, Seijin Kobayashi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.