Task Context Shapes Component Roles in Language Models
Task context changes which facts shared language-model heads use and how they influence later computation.
The paper compares fixed language-model components across tasks using identical graph facts and measures both later-layer influence and answer importance. Task changes redistribute influence in five open-weight models spanning four families and 1.5 to 14 billion parameters. In Gemma-2-9B and Qwen3-14B, an activation edit switches which fact the model reads while the written instruction stays fixed, and the same heads then influence later components differently. Linear response maps matched to the edited condition reduce mean squared prediction error by 53.6% in Gemma and 5.0% in Qwen versus the average of the two maps.
- Shared attention heads change influence when the task changes.
- Five open-weight models from 1.5B to 14B parameters were tested.
- An activation edit in Gemma-2-9B and Qwen3-14B switches which fact is read.
- Condition-matched response maps cut prediction error 53.6% in Gemma and 5.0% in Qwen.
- AI models
- Gemma-2-9BQwen3-14B
From the original article
1 min read at arxiv.orgLarge language models reuse the same attention heads across tasks, but identifying these shared heads does not explain what they do differently in each task. A natural source of inspiration is task-dependent interactions among shared brain regions, which motivate a network view of language-model computation.
By Sovesh Mohapatra · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.