SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
SWE-Serve benchmarks agents on 53 SGLang production inference tasks and exposes a gap versus end-to-end correctness.
SWE-Serve is a benchmark of 53 repository-grounded tasks taken from recent production changes to SGLang and grouped into six inference-engineering families. Each task runs on CPU or a single H100 and is scored with hidden functional and regression tests, plus end-to-end serving tests and performance gates where applicable. Across 11 models and 31 model-effort configurations, the best configuration reaches 75% mean pass@1. On 19 tasks with end-to-end coverage, serving tests reject about one-third of otherwise passing patches, with 45.9% under the verifier versus 69.4% when those tests are excluded.
- Fifty-three repository tasks span six SGLang inference-engineering families.
- Tasks run on CPU or one H100 with hidden functional and regression tests.
- Best of 31 model-effort configurations reaches 75% mean pass@1.
- On 19 end-to-end tasks, pass rate falls from 69.4% to 45.9%.
From the original article
1 min read at arxiv.orgWe introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs.
By Jennifer Williams · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.