SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
SWE-Serve benchmarks agents on 53 SGLang production inference tasks and exposes a gap versus end-to-end correctness.
SWE-Serve is a benchmark of 53 repository-grounded tasks taken from recent production changes to SGLang and grouped into six inference-engineering families. Each task runs on CPU or a single H100 and is scored with hidden functional and regression tests, plus end-to-end serving tests and performance gates where applicable. Across 11 models and 31 model-effort configurations, the best configuration reaches 75% mean pass@1. On 19 tasks with end-to-end coverage, serving tests reject about one-third of otherwise passing patches, with 45.9% under the verifier versus 69.4% when those tests are excluded.
- Fifty-three repository tasks span six SGLang inference-engineering families.
- Tasks run on CPU or one H100 with hidden functional and regression tests.
- Best of 31 model-effort configurations reaches 75% mean pass@1.
- On 19 end-to-end tasks, pass rate falls from 69.4% to 45.9%.
Full article238 words · extracted from arxiv.org · click to collapse
We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs. Existing benchmarks provide limited coverage of production inference engineering: repository-level software engineering benchmarks do not target inference, while general terminal-agent benchmarks include only a few inference tasks. Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation. SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families. Each task executes on either CPU or a single GPU (H100) and is evaluated with hidden functional and regression tests, including, where applicable, end-to-end (E2E) serving tests and calibrated performance gates. Executable no-op and oracle controls, adversarial verifier review, and closed-book execution support task validity and evaluation integrity. Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1. SWE-Serve exposes a substantial gap between completing tasks locally and achieving production correctness. On 19 tasks with end-to-end coverage, model-serving E2E tests reject roughly one-third of patches that pass every other test (45.9% under the verifier versus 69.4% with E2E tests excluded from scoring), with pass rate increasing for each model's best-performing configuration. By making the production correctness gap directly measurable, SWE-Serve enables the field to track whether future agents move beyond completing tasks locally to achieving production correctness.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.26777