Lexical reference frames read LM hidden states without a lens
Original title: Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics
LexFrame[1] reaches 0.97 coherence on SmolLM3-3B and OLMo3-7B, versus 0.30 and 0.35 for fixed lenses.
The paper proposes lexical reference frames that read a hidden state through the model's remaining layers, without fitting a lens to the output. Fixed-direction lenses cannot exceed the square root of native-frame coherence, 0.30 and 0.35 at mid-depth of SmolLM3-3B and OLMo3-7B. The first-order frame, LexFrame[1], reaches 0.97 on both and reads within 0.16 nats of the model output. Its coefficients attribute and steer predictions more selectively than attribution patching.
- LexFrame[1] stays within 0.16 nats of the model's output.
- Confident readings could skip 0.45 and 0.42 of the layers.
- Fixed lenses skip almost none of those layers.
- Its lexical basis steers one prediction while sparing other tokens.
From the original article
1 min read at huggingface.co
The vocabulary of a language model is its native semantics: the embedding and the unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods either reuse a fixed lens or treat the states as a point cloud, and neither reads a state the way the model does.
By Jian Gu, Hongyu Zhang, Chunyang Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.