A mechanistic study of language model introspection
Researchers identify attention-head circuits that let language models detect and localize hidden activation interventions.
Large language models can sometimes report perturbations to internal activations even when the prompt gives no evidence of an intervention. With input text held fixed, the authors inject a concept vector at one of ten token positions or apply no intervention, then ask the model to name the position. Across three model families, middle-layer gate heads influence whether a change is reported, while later router heads help select the position. Concept vectors localized more accurately produce stronger gate-head responses, associated with better-aligned key and value changes in QK and OV computations.
- Fixed-text task injects a concept vector at one of ten positions or none.
- Middle-layer gate heads control whether a change is reported.
- Later-layer router heads help choose which position to report.
- Suppressing gate heads blocks reports even when routers hold location information.
- Better-localized concepts drive stronger, better-aligned gate-head QK and OV responses.
From the original article
1 min read at arxiv.orgLarge language models (LLMs) can sometimes report perturbations to their internal activations---even when the input provides no evidence that an intervention occurred. How do models detect and localize such internal changes? We study this question using a controlled task that keeps the input text fixed.
By Jiahong Zou · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.