From Parameters to Answers: How LLMs Retrieve and Use Their Internal Knowledge
Interpretability study traces how Qwen, Llama, and Gemma route query information and internal knowledge across layers when answering questions.
Researchers used layerwise interventions on hidden states to separate query-routing signals from target knowledge in language models, testing Qwen, Llama, and Gemma on country-continent questions with varied answer types. A pair-conditioned request direction strengthens before interventions alter downstream knowledge, opening a causal window while answer-supporting content is still forming. Trajectories differ by model: Gemma shows a partially overlapping mid-layer routing profile, while Llama has no sustained routing-effect window under the same gates.
- Separates early readability, causal steering, and later content dependence.
- Layerwise interventions isolate request-direction effects from knowledge use.
- Routing behavior differs across Qwen, Llama, and Gemma.
- Pair-conditioned request direction retains a late effect in Qwen.
From the original article
1 min read at arxiv.orgHow does a language model's dependence on query-routing information and target knowledge change as it answers a question? We study this question through layerwise interventions on the hidden state at the end of the question. Across Qwen, Llama, and Gemma, we compare country-continent questions with noun, adjective, and code answers while keeping several fitted measurements distinct.
By Wenkang Wei · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.