IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
IntBMoE is a block-conditioned Mixture-of-Experts architecture achieving full participation with sparse execution, deployed in AMap's recommendation system.
IntBMoE is a block-conditioned Mixture-of-Experts architecture that decouples participation, execution, and materialization using a learned codebook. The method achieves full expert participation while maintaining sparse execution through Dual-Path Residual Gating. The system is deployed in AMap's recommendation system serving hundreds of millions of users with 2.4% improvement in UVCTR.
- IntBMoE decouples participation, execution, and materialization in Mixture-of-Experts architectures.
- Block-conditioned MoE achieves full expert participation with sparse execution.
- Deployed in AMap's generative recommendation system serving hundreds of millions of users.
- Achieved 2.4% relative UVCTR gain in online A/B testing under 60ms latency.
Full article263 words · extracted from huggingface.co · click to collapse
Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.21346