Small open-weight coding workers trail direct coding on checks
Original title: Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?
In one author's studies, a Qwen3-Coder worker passed 0/45 tasks while direct coding passed 45/45 on the same suite.
A retrospective report covers the author's coding-agent work from July to October 2026 across clients and serving stacks including vLLM, SGLang, llama.cpp, and MLX. Nominally compatible endpoints still differed in tool-call formatting, turn control, context, cache reuse, and reliability. Evidence includes 4,165 gateway turns, 623 debug jobs, and delegation studies with a fixed Claude Opus 5.5 parent and hidden behavioral checks. Direct coding beat quantized workers on verified success and list-price-equivalent cost; the studies are not pooled and do not rank models or runtimes.
Unconfirmed: Figures come from one engineer's sessions and are not a pooled ranking of models or runtimes.
- Cold-prefix median time to first token was about 69.4 seconds versus 2.8 seconds.
- One 45-trial study: direct coding 15/15; Qwen3-Coder 30B-A3B 12/15 and 14/15.
- Second suite: direct coding 45/45; Nemotron Cascade 2 worker 12/45.
- Counting 16 subscription-limit stops as failures gives 37/45 and 8/45.
- A Qwen3-Coder worker passed 0/45; 19 renamed-parameter rejects were not counted as successes.
From the original article
1 min read at huggingface.co
Making an open-weight language model generate text is not the same as making it finish coding work. This retrospective empirical systems experience report follows my engineering work from July to October 2026 across coding clients, model-specific serving recipes, vLLM, SGLang, llama. cpp, and MLX.
By Vaibhav Jain · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.