GearUp hides serverless ML loading using DAG slack time
On NVIDIA L40S GPUs, median latency falls 64.5% and 75.9% versus SMIless on two workflows.
Stories per day · 30 days
4 stories, newest first · one card per story
Stories tagged #inference-serving
EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models
EAServe raises multimodal LLM serving goodput by controlling encode, prefill placement, and GPU sharing.
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
SWE-Serve benchmarks agents on 53 SGLang production inference tasks and exposes a gap versus end-to-end correctness.
PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving
PACE cuts perceived latency in retrieval-augmented dialogue serving via cascading routing and filler control, reaching 0.41s P95 under load.