GearUp hides serverless ML loading using DAG slack time
Original title: GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading
On NVIDIA L40S GPUs, median latency falls 64.5% and 75.9% versus SMIless on two workflows.
GearUp is a serverless inference system for machine-learning workflows expressed as DAGs. It profiles each node's library, weight, and CUDA-kernel loading and schedules that work in slack time so initialization is hidden behind upstream execution. Idle initialized instances are reused, and a traffic-adaptive keep-alive policy trades memory against cold-start risk. On AMBER Alert and FINRA-ML with NVIDIA L40S GPUs, median end-to-end latency drops 64.5% and 75.9% versus SMIless, with median speedups of 3.1× and 10.0× over cold execution.
- GPU and host memory use fall by up to 83.9% and 94.6% versus InstaInfer.
- GearUp reports 4.2× strong scaling on four workers.
- Eager pre-loading is the InstaInfer comparison baseline.
- Loading stays off the critical path at lower memory cost than DAG-agnostic preload.
- Vendors
- NVIDIA
- Products
- GearUpNVIDIA L40S
From the original article
1 min read at arxiv.orgServerless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path.
By Qi Qi · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.