Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization
Researchers personalize mixture-of-experts LLMs and localize distinct personalization features in specific experts.
The paper argues that mixture-of-experts LLMs are easier to personalize and interpret than dense models because sparse routing reduces polysemantic neurons. The authors adapt three unnamed open-weight MoE models and train crosscoders on paired personalized and non-personalized activations to find experts tied to distinct personalization features. Personalized MoEs perform competitively with dense LLMs while activating only a fraction of parameters and show lower polysemanticity. Causal tests indicate the identified latents can recover personalized behavior and shift a non-personalized model toward it.
- Three open-weight MoE LLMs were adapted on a large personalization dataset.
- Crosscoders learned personalized latents from paired model activations.
- Personalized MoEs matched dense models while activating fewer parameters.
- Causal interventions recovered personalized behavior by boosting associated experts.
From the original article
1 min read at huggingface.co
Personalization of Large Language Models (LLMs) is increasingly important for user-centric applications, yet prior methods mainly adapt dense LLMs, whose polysemantic neurons, each encoding multiple unrelated concepts, hinder localizing the components behind personalization, and thus understanding and controlling it.
By Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.