Language-subspace interventions fail to fully capture language information in LLMs
Original title: Beyond Accuracy: What Language-Subspace Interventions Capture, Remove, and Improve
Language-subspace interventions improve multilingual accuracy by only 1.60 points but sacrifice response-language fidelity in Qwen2.5-7B.
We investigate language-subspace interventions on MATH and MGSM across four LLMs. Subspaces capture translation variation but projecting them out does not remove all language information as probes still identify language. The best intervention improves MGSM accuracy slightly but damages language fidelity. Subtle prompt changes can reproduce gains with damage.
- Subspaces show similar patterns across model depth
- Probes classify language almost perfectly from projected activations
- One intervention improves joint accuracy by 1.60 points but sacrifices response-language fidelity
- Subtle changes in prompt alone reproduce accuracy gains albeit with damage to fidelity
- Tags
- multilingual
- AI models
- Qwen2.5-7B
From the original article
1 min read at arxiv.orgLanguage-subspace interventions aim to improve multilingual reasoning by suppressing language-related components in mid-layer activations. Using MATH and MGSM across four open-weight LLMs, we ask three questions: do estimated language subspaces capture translation-associated variation? Does projecting them out remove language information?
By Shaanil Punglia · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.