Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
Local serving stacks can distort tool-use scores, including false 0% fidelity for rejected models.
Researchers show that local serving stacks can change measured tool-call fidelity independently of the model. In Ollama, a static template flag gates the default tools request: some models return text or native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. If those rejections are not stored as structured failures, downstream analysis can report 0% fidelity. Keeping a text tool list alongside the native channel recovered fidelity for accepted models, but a text-only protocol reduced it for Llama-3.2. Probes of Ollama, llama.cpp, vLLM, and SGLang diverged, constrained decoding could fail to terminate, and pooling methods differed by about 55 points.