Encoded Early, Used Late: Where Transformers Begin to Act on an Inferred Partner's Expertise
Probing finds transformers represent an inferred dialogue partner's expertise in early layers long before it causally influences output.
Using ExpertCollab, a corpus of multi-turn research-planning dialogues between model-played personas at four expertise levels, researchers show that a partner's inferred expertise is most decodable in early transformer layers and decays to near chance before the network's midpoint. Counterfactual patching reveals that injecting the expertise difference at peak decodability barely changes a fixed late-layer readout, while injection past the midpoint propagates almost completely. The result bounds where readout or steering of partner-conditioned behavior must intervene, demonstrated on a single model with a synthetic corpus.
- Partner expertise is most decodable in early transformer layers
- Early-layer information is largely causally inactive at late-layer readouts
- Injection past the network midpoint propagates over an order of magnitude more
- Demonstrated on one model using the synthetic ExpertCollab corpus
From the original article
1 min read at huggingface.co
A transformer can make an attribute linearly decodable in its residual stream at a depth where that attribute does not yet influence the output. This gap between where information is readable and where it is used has been shown for attributes stated directly in the input. We ask whether it also holds for an attribute the model must infer gradually over a conversation, namely how expert its dialogue partner is.
By Mika Okamoto, Gabriele Sarti · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.