Hybrid models suppress attention retrieval unless recurrence updates pause
Original title: Competing Learning in Hybrid Architectures
A staged pretraining procedure that freezes recurrence mid-training improves loss, reasoning, and retrieval on models up to 1B parameters.
Authors describe Competing Learning in hybrid language models that mix linear-time recurrence and softmax attention. Recurrence learns retrieval faster and suppresses the learning signal for attention. Stopping recurrence weight updates mid-training lets attention learn retrieval. On models up to 1B parameters, staged pretraining improved language-modeling loss, common-sense reasoning, and retrieval across context lengths.
Unconfirmed: Performance gains are reported by the paper's authors and are not independently confirmed here.
- Recurrence compresses context into a fixed-size state but lags attention on retrieval.
- Attribution shows recurrence usually learns retrieval before softmax attention.
- Freezing recurrence updates mid-training restores attention's retrieval learning.
- Reported gains cover loss, common-sense reasoning, and retrieval tasks.
From the original article
1 min read at huggingface.co
Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval.
By Xingyu Shen, Yingfa Chen, Ziqing Qiao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.