Competing learning found in hybrid recurrence-attention models
Original title: Competing Learning in Hybrid Architectures
Stopping recurrence updates mid-training lets attention learn retrieval; staged pretraining improved models up to 1B parameters.
The paper describes Competing Learning in hybrid language models that mix linear-time recurrence with softmax attention. Under typical data, recurrence learns retrieval faster and suppresses the learning signal for attention, so attention often fails to learn retrieval. Freezing recurrence weights partway through training lets attention acquire retrieval, and a staged pretraining procedure improved language-modeling loss, common-sense reasoning, and retrieval tasks on models up to 1B parameters.
- Recurrence compresses context into a fixed-size state but lags attention at retrieval.
- When one mixer learns a function, it suppresses that signal for other layers.
- Layer attribution shows recurrence usually wins retrieval under typical data.
- Gains cover common-sense reasoning and retrieval across context lengths.
From the original article
1 min read at arxiv.orgMany modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval.
By Xingyu Shen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.