Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
Local serving stacks can distort tool-use scores, including false 0% fidelity for rejected models.
Researchers show that local serving stacks can change measured tool-call fidelity independently of the model. In Ollama, a static template flag gates the default tools request: some models return text or native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. If those rejections are not stored as structured failures, downstream analysis can report 0% fidelity. Keeping a text tool list alongside the native channel recovered fidelity for accepted models, but a text-only protocol reduced it for Llama-3.2. Probes of Ollama, llama.cpp, vLLM, and SGLang diverged, constrained decoding could fail to terminate, and pooling methods differed by about 55 points.
- Ollama's default tools request is gated by a static per-model template flag.
- Phi-3 and Gemma-3 are rejected before inference, which can look like 0% fidelity.
- A uniform text protocol lowers fidelity for Llama-3.2, which supports native tool calls.
- Ollama, llama.cpp, vLLM, and SGLang handle the same request differently.
- Turn-pooled and per-instance estimates differed by up to about 55 points.
From the original article
1 min read at arxiv.orgA coding agent must emit a valid tool call--a parseable invocation of a tool in the provided schema--before the harness can execute its chosen action. We study how local serving stacks affect this protocol step and show that measured outcomes can depend on the serving layer rather than model behavior alone.
By Lijuan Tang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.