Skip to content
NewA calmer, faster ZeroHour

EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models

arXiv cs.AI / cs.LG / cs.CLby Kunxiong Zhu ()1 min read5 · 4 reads
AI summary · grok-4.7Summaries by Vani AI

EAServe raises multimodal LLM serving goodput by controlling encode, prefill placement, and GPU sharing.

EAServe treats the Encode stage as the control point of the Encode-Prefill-Decode pipeline for multimodal LLMs. Its runtime combines load-adaptive micro-batching, rate-controlled partial offload onto a co-resident prefill worker, and dynamic SM partitioning. Hybrid Auto Selection prunes GPU allocations using per-stage capacity profiles and tunes encode batch size and offload ratio with TPE Bayesian optimization. Across image, video, and audio models it reports up to 4.3x goodput versus NVIDIA Dynamo and 1.7x versus vLLM under the same SLOs.

Key facts
  • Per-request encode leaves GPUs idle and starves prefill and decode.
  • Runtime adds micro-batching, partial prefill offload, and dynamic SM partitioning.
  • HAS profiles capacity, then tunes batch size and offload ratio.
  • Up to 4.3x goodput versus NVIDIA Dynamo and 1.7x versus vLLM.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Disaggregating the two stages, Prefill and Decode, onto separate GPU pools is now a standard optimization for (text-only) LLM serving. However, multimodal LLMs (MLLMs), which add a third phase, Encode, pose new challenges for resource allocation. Encode turns images, video, or audio into embeddings that the language model can consume, yielding a three-stage Encode-Prefill-Decode (EPD) pipeline.

arXiv cs.AI / cs.LG / cs.CL

By Kunxiong Zhu · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.