42
30
30
42
45
57
42
55
Large Language Models Develop Belief State Geometry In-Context
Probing six open-source LLMs on HMM-generated data shows belief states are linearly decodable (R² 0.83–0.99), suggesting in-context learning approximates Bayesian prediction.
Researchers prompted six open-source LLMs with data from 40 hidden Markov models selected for non-trivial belief structure and probed residual-stream activations for belief states (posteriors over hidden states). Belief states were linearly decodable with peak R² values of 0.83–0.99 across HMM/LLM combinations, spanning early to late layers. Patching and steering the probe-identified subspace preserved downstream prediction quality while control interventions degraded performance substantially, establishing functional relevance. The results provide representation-level evidence that in-context learning approximates optimal Bayesian prediction over a context-inferred generative model.
45
Friday Squid Blogging: The Symbiotic Relationship Between the Bobtail Squid and a Particular Microbe
30
55
30
45
30
30
30
30
55
45
55
30
30
45
42
30
30
45
42
45
30
42
42
42
30
30
45
42
30
30
30