Large Language Models Develop Belief State Geometry In-Context
Probing six open-source LLMs on HMM-generated data shows belief states are linearly decodable (R² 0.83–0.99), suggesting in-context learning approximates Bayesian prediction.
Researchers prompted six open-source LLMs with data from 40 hidden Markov models selected for non-trivial belief structure and probed residual-stream activations for belief states (posteriors over hidden states). Belief states were linearly decodable with peak R² values of 0.83–0.99 across HMM/LLM combinations, spanning early to late layers. Patching and steering the probe-identified subspace preserved downstream prediction quality while control interventions degraded performance substantially, establishing functional relevance. The results provide representation-level evidence that in-context learning approximates optimal Bayesian prediction over a context-inferred generative model.
- Belief states linearly decodable from residual stream, peak R² 0.83–0.99
- Six open-source LLMs tested on 40 HMMs with non-trivial belief structure
- Patching/steering the subspace preserves prediction; controls degrade it
- Evidence ICL approximates optimal Bayesian prediction over inferred generative model
From the original article
1 min read at arxiv.orgLarge language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly understood.
By Daniel Balcells · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.