Characterizing High Bandwidth Flash for LLM Serving
High-bandwidth flash cuts LLM serving completion time by 36-87% when scheduling preserves flash write lifetime.
The paper evaluates high-bandwidth flash as extra capacity for LLM weights and KV caches, whose access cost and write endurance complicate serving. An HBM-HBF-host hierarchy with buffered cache-aware scheduling is studied via trace-driven simulation. The fastest HBF-augmented systems reduce completion time by 36.1-87.0% versus HBM-only systems, with modeled energy savings up to 55.8%, though some light workloads use more energy. Buffered scheduling extends estimated HBF write lifetime from 4.77 to 14.82 years in the evaluated configuration.
- HBM-HBF-host storage and buffered cache-aware scheduling studied with trace-driven simulation.
- Fastest HBF setups cut completion time 36.1-87.0% versus HBM-only serving.
- Modeled energy savings reach 55.8%, but some light workloads use more energy.
- Scheduling extends estimated HBF write life from 4.77 to 14.82 years.
From the original article
1 min read at arxiv.orgLarge language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse.
By Zack Yu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.