Competing learning found in hybrid recurrence-attention models
Stopping recurrence updates mid-training lets attention learn retrieval; staged pretraining improved models up to 1B parameters.
Stories per day · 30 days
Relatedhybrid-architectures2language-models2recurrence2fft1kernel-methods1numerical-integration1pretraining1quasi-monte-carlo1rank-1-lattice1retrieval1
3 stories, newest first · one card per story
Stories tagged #softmax-attention
Hybrid models suppress attention retrieval unless recurrence updates pause
A staged pretraining procedure that freezes recurrence mid-training improves loss, reasoning, and retrieval on models up to 1B parameters.
Subgroup Rank-1 Lattice for Practical High-dimensional Black-box Integral Approximation
A subgroup rank-1 lattice method speeds high-dimensional black-box integral estimates, including softmax attention, via FFT correlations.