Competing learning found in hybrid recurrence-attention models
Stopping recurrence updates mid-training lets attention learn retrieval; staged pretraining improved models up to 1B parameters.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #hybrid-architectures
Hybrid models suppress attention retrieval unless recurrence updates pause
A staged pretraining procedure that freezes recurrence mid-training improves loss, reasoning, and retrieval on models up to 1B parameters.