Cross-Lingual Alignment for Decoder-Only Models using MoE Routers
Aligning MoE router outputs improved cross-lingual transfer in four open-source decoder-only models.
The paper adapts cross-lingual contrastive learning to decoder-only LLMs by aligning mixture-of-experts router outputs rather than applying an auxiliary loss on hidden states. Router outputs are easier to pool across tokens, supporting sequence-level cross-lingual comparison despite mismatched tokenization. Controlled continual pre-training on four open-source MoEs aligned underlying hidden representations and improved multilingual performance on the authors' evaluation suite.
- Alignment targets MoE router outputs instead of hidden states.
- Router outputs pool more reliably across tokens for sequence-level comparison.
- Continual pre-training on four open-source MoEs also aligned hidden representations.
- The routing loss improved multilingual scores on a diverse evaluation suite.
From the original article
1 min read at huggingface.co
Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is not possible in decoder-only LLMs because of varying multilingual tokenization. However, a growing amount of research suggests that even in LLMs, higher cross-lingual representational alignment leads to improved cross-lingual transfer.
By Lucas Bandarkar, Clark Peng, Ahmed Haj Ahmed · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.