Meta Expert shares MoE experts across layers to cut redundancy
Original title: Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
With activation and granularity fixed, it matches vanilla MoE using 70–80% of the experts.
The paper argues experts in different layers of hierarchical mixture-of-experts models store similar content and proposes sharing Meta Experts across layers. Comparisons find lower parameter redundancy, lower cross-entropy loss, and higher downstream accuracy than local experts. With activated count and expert granularity fixed, the design matches vanilla MoE using 70–80% of the experts. The authors also say Meta Experts give hidden states more flexible residual pathways.
Unconfirmed: Efficiency and accuracy figures are the authors' reported findings.
- Activated expert count and granularity are held fixed.
- Authors report lower cross-entropy and higher task accuracy.
- Theory claims more flexible residual pathways improve retrieval.
From the original article
1 min read at huggingface.co
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content.
By Ermo Hua, Xiangyu Hong, Baiting Wu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.