Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
Pythia encodes task information in hidden states before its unembedding readout can express it.
The paper shows that language-model hidden states can encode task information before the unembedding readout expresses it, so timelines based only on generated tokens conflate encoding with expression. Parameter-trajectory crosscoding fits one sparse dictionary to every token's readout row across checkpoints using weights alone. On Pythia, the readout reorganizes in a narrow window at 0.4–1.1% of pretraining, when WordNet categories become decodable from readout rows, while linear probes already recover task information the outputs do not yet show. Raising only the readout learning rate makes that gap close sooner with little effect on final loss.
- Output timelines mix what a model encodes with what its readout can express.
- Parameter-trajectory crosscoding reads unembedding rows from checkpoint weights alone.
- WordNet categories become decodable at 0.4–1.1% of Pythia pretraining.
- Raising the readout learning rate closes the encode-versus-express gap sooner.
From the original article
1 min read at huggingface.co
When does a language model acquire a capability? Developmental studies usually answer this by tracking when models start producing the correct token. But that prediction passes through the unembedding matrix, or readout, which is itself still being learned during pretraining.
By Matteo He, William F. Shen, Alex Iacob · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.