Speculative Decoding in vLLM on AMD GPUs
vLLM benchmarks speculative decoding on AMD Instinct MI300X and MI355X GPUs across five drafting methods including EAGLE-3 and native MTP.
The vLLM project documents draft-and-verify speculative decoding support for AMD GPUs via ROCm, comparing native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark drafting approaches. Output-token throughput effects varied with drafting method, proposal length, model family, draft checkpoint, workload, and acceptance behavior. The post also covers how to enable each method plus practical tuning and observability considerations.
- Speculative decoding lets a lightweight drafter propose tokens verified in one target-model pass
- Five drafting methods evaluated: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, DSpark
- Tests ran on AMD Instinct MI300X and MI355X using the ROCm software stack
- Gains vary by model family, workload, and acceptance behavior, requiring per-deployment tuning
From the original article
13 min read at vllm.aiTL;DR: Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior.
By ankitg12 · vllm.ai
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.