Pretraining Latent Information Feedback Transformers with Teacher Supervision
LIFT pretrains Transformers to recycle latent state and beats token-matched baselines from 135M to 1B.
LIFT (Latent Information Feedback Transformer) lets language models pass latent state across generation steps instead of relying only on the decoded token. Each input token is paired with a precomputed state derived from an off-the-shelf pretrained model's next-token distribution, and a small parameter extension is trained to predict both the next token and the next state, keeping pretraining parallel. At inference the model's own predicted states are fed back with overhead that shrinks as model size grows. On models from 135M to 1B parameters, LIFT beats standard Transformers and baselines on language modeling, reasoning, and procedural tasks under a token-matched budget, and matches or exceeds compute-matched Transformers. On a state-tracking task, a tiny LIFT outperforms same-size Transformers trained on eight times more data.
- LIFT feeds predicted latent state from deep layers back to shallower ones.
- Teacher states come from an off-the-shelf LM's next-token distribution.
- Pretraining stays parallel because input states are precomputed.
- Models from 135M to 1B beat standard Transformers under token-matched budgets.
- A tiny LIFT beats same-size Transformers trained on 8x more state-tracking data.
From the original article
1 min read at arxiv.orgTransformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative continuations.
By Dor Tirosh · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.