User Model Extraction via Belief Self-Distillation
Belief Self-Distillation reads and rewrites an LLM's user beliefs, changing refusals with the request fixed.
The paper introduces Belief Self-Distillation, which learns a compact user representation from a frozen large language model acting as its own teacher, using natural conversations and no external annotations. Unlike conventional probing, the representation can be written back so its causal role can be tested, and it yields stronger interventions than matched hidden-state steering across multiple model families. The authors find that refusal depends on inferred user intent: changing that belief alters refusal while the request stays fixed. Independently trained models also converge on a shared geometry for representing users.
- BSD distills user beliefs from conversations without external labels.
- The representation can be decoded and written back for causal tests.
- Editing inferred user intent changes refusal with the request fixed.
- Separately trained LLMs share a geometry for user representations.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) implicitly infer attributes of their users and adapt their behavior accordingly, yet these beliefs remain difficult to inspect and causally manipulate. We introduce Belief Self-Distillation (BSD), a unified read-write framework that bridges linear and causal probing by learning a compact user representation that can be both decoded and written back into the model.
By Ali Holmov · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.