Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Flash-dLLM accelerates diffusion LLM inference with I/O-aware KV caching, up to 11 times faster than Elastic-Cache.
Flash-dLLM is a training-free inference framework for diffusion large language models that treats GPU memory I/O as the main bottleneck when KV caching and parallel decoding are used together. An I/O-aware fused KV-cache kernel reduces redundant memory traffic, and a cache-driven draft-and-verify strategy lets the same model draft and check tokens without an auxiliary drafter. The authors report better speed and memory use than prior dLLM accelerators, including 5.1 times and 11.0 times speedups over Elastic-Cache on GSM8K and HumanEval, while preserving generation quality and scaling to longer sequences and larger batches.
- Training-free inference acceleration for diffusion large language models.
- A fused KV-cache kernel reduces redundant GPU memory movement.
- The same model drafts and verifies tokens without an auxiliary model.
- Reports 5.1 times and 11 times speedups over Elastic-Cache on GSM8K and HumanEval.
From the original article
1 min read at huggingface.co
Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive LLMs by enabling non-autoregressive text generation. However, their practical deployment remains limited by inefficient inference, largely due to the absence of effective Key-Value (KV) caching and scalable parallel decoding mechanisms.
By Quan Nguyen-Tri, Mukul Ranjan, Zhiqiang Shen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.