MAEP prunes MoE multimodal experts by modality in one pass
Original title: Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models
The training-free method cuts about 27 GB of BF16 expert weights so an about 58 GB model fits a 40 GB GPU.
The paper proposes MAEP, a training-free expert-pruning framework for mixture-of-experts multimodal large language models. It scores experts by modality in a single forward pass, using text-side importance to preserve experts and vision-side importance, including low-attention vision tokens, as a pruning signal. On three MoE MLLMs, the authors report better results than modality-agnostic baselines on most multimodal benchmarks while staying competitive on text tasks. Code is released.
Unconfirmed: Compression and benchmark gains are author-reported and not independently confirmed in this item.
- Experts are over 90% of model size in the cited MoE MLLMs.
- InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B use 54 GB of about 58 GB in BF16 for experts.
- MAEP uses Delta-H Output to measure output change when an expert is removed.
- Reported savings are about 27 GB, about 47% of total memory.
- The authors say the reduced footprint fits a single 40 GB GPU.
- Products
- MAEP
- AI models
- InternVL3.5-30B-A3BQwen3-VL-30B-A3B
From the original article
1 min read at huggingface.co
Mixture-of-Experts (MoE) enables multimodal large language models (MLLMs) to scale capacity efficiently, but expert parameters dominate memory even though only a few experts are activated per token: they account for over 90% of model size in modern MoE-MLLMs (e. g. , 54 GB out of about 58 GB in BF16 for InternVL3.
By Donggeun Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.