Inference serving under scrutiny: hidden evaluation confounds in local stacks and a new agent benchmark for production serving code

Two arXiv studies (both 2026-09-22) put the serving layer itself in focus: local stacks like Ollama can distort tool-use fidelity scores (including false 0% for rejected models), while the new SWE-Serve benchmark shows end-to-end serving tests reject roughly…

Two separate arXiv papers (cs.AI/cs.LG/cs.CL, both dated 2026-09-22) examine the inference serving stack from complementary angles. The first, 'Measuring the Serving Stack Instead of the Model,' shows that local serving stacks can change measured tool-call fidelity independently of the model. In Ollama, a static per-model template flag gates the default tools request: some models return text or native tool_calls, while Phi-3 and Gemma-3 are rejected before inference; if those rejections are not recorded as structured failures, downstream analysis can report a false 0% fidelity. Keeping a text tool list alongside the native channel recovered fidelity for accepted models, but a uniform text-only protocol reduced fidelity for Llama-3.2. Probes of Ollama, llama.cpp, vLLM, and SGLang diverged on identical requests, constrained decoding could fail to terminate, and turn-pooled versus per-instance estimates differed by up to about 55 points. The second paper introduces SWE-Serve, a benchmark of 53 repository-grounded tasks drawn from recent production changes to SGLang and grouped into six inference-engineering families. Tasks run on CPU or a single H100 and are scored with hidden functional and regression tests plus end-to-end serving tests and performance gates. Across 11 models and 31 model-effort configurations, the best reaches 75% mean pass@1; on the 19 tasks with end-to-end coverage, serving tests reject about one-third of otherwise passing patches, with 45.9% pass rate under the verifier versus 69.4% when those tests are excluded. Together, the studies suggest both model evaluations and engineering-agent evaluations are sensitive to serving-layer choices and test methodology. The papers do not contradict each other; they cover different problems.

• Paper 1 (2026-09-22): local serving stacks can distort tool-call fidelity independently of the model; study probes Ollama, llama.cpp, vLLM, and SGLang, which handle the same request differently.
• Ollama's default tools request is gated by a static per-model template flag; Phi-3 and Gemma-3 are rejected before inference, which can appear as 0% fidelity if rejections are not stored as structured failures.
• Keeping a text tool list alongside the native channel recovered fidelity for accepted models, but a text-only protocol lowered fidelity for Llama-3.2, which supports native tool calls.
• Constrained decoding could fail to terminate; turn-pooled and per-instance fidelity estimates differed by up to about 55 points.
• Paper 2 (2026-09-22): SWE-Serve contains 53 repository-grounded tasks from recent production changes to SGLang, spanning six inference-engineering families.
• SWE-Serve tasks run on CPU or a single H100 and are scored with hidden functional and regression tests, plus end-to-end serving tests and performance gates where applicable.
• Across 11 models and 31 model-effort configurations, the best SWE-Serve configuration reaches 75% mean pass@1.
• On the 19 SWE-Serve tasks with end-to-end coverage, serving tests reject about one-third of otherwise passing patches: 45.9% under the verifier versus 69.4% when those tests are excluded.