Specter speeds single-GPU MoE inference via speculative expert offloading
Original title: Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
A quantized on-device draft model predicts future experts, cutting time per output token by 1.50x on average.
Specter is an inference system for mixture-of-experts models that do not fit in one accelerator. Quantization lets a lightweight draft model stay on device and forecast expert use several steps ahead, so expert transfers and execution can be planned before verification. Under single-request inference it reports an average 1.50x speedup in time per output token, up to 1.88x, versus the prior state-of-the-art system. Code is released for commodity single-GPU use.
- Offloading experts to host memory leaves PCIe transfers on the critical path.
- The draft model is meant to preserve high-fidelity expert activation patterns.
- Reported speedup is up to 1.88x time per output token.
- Code is at github.com/wwwwwf/specter_asplos.
From the original article
1 min read at huggingface.co
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading.
By Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.