IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
IntBMoE is a block-conditioned Mixture-of-Experts architecture achieving full participation with sparse execution, deployed in AMap's recommendation system.
IntBMoE is a block-conditioned Mixture-of-Experts architecture that decouples participation, execution, and materialization using a learned codebook. The method achieves full expert participation while maintaining sparse execution through Dual-Path Residual Gating. The system is deployed in AMap's recommendation system serving hundreds of millions of users with 2.4% improvement in UVCTR.
- IntBMoE decouples participation, execution, and materialization in Mixture-of-Experts architectures.
- Block-conditioned MoE achieves full expert participation with sparse execution.
- Deployed in AMap's generative recommendation system serving hundreds of millions of users.
- Achieved 2.4% relative UVCTR gain in online A/B testing under 60ms latency.
- Vendors
- AMap
- Products
- IntBMoEDreamX-Rec
- Organizations
- AMap
- AI models
- IntBMoE
From the original article
1 min read at huggingface.co
Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost).
By Ran Cheng, Longfei Xu, Zheng Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.