Some official Ollama low-bit GGUF tags fail code tasks despite intact weights
Original title: Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them
Seven of 305 official Ollama tags solved at most six of 164 HumanEval+ tasks.
Researchers tested 305 code-capable GGUF tags across 15 lines in the official Ollama library and 22 Hugging Face community files. None of the examined registries documented functional testing. Seven official tags, 2.3 percent, solved at most six of 164 HumanEval+ tasks; all were low-bit quantizations of models with at most four billion parameters. Five rebuilt files matched registry tensors, ruling out corruption. An importance matrix raised three Qwen2.5-Coder-3B builds from zero to 76–113 solved tasks. They release quantcheck and the audit data.
- Seven failing official tags were low-bit quantizations of models of at most 4B parameters.
- Five registry files matched uncalibrated rebuilds tensor for tensor, so none was corrupted.
- None of 66 tags that passed the initial screen was non-functional.
- The same uncalibrated recipe left Qwen2.5-Coder-1.5B functional.
- Vendors
- Ollama
- Products
- GGUFquantcheckllama.cpp
- Organizations
- Hugging Face
- AI models
- Qwen2.5-Coder-3BQwen2.5-Coder-1.5B
From the original article
1 min read at huggingface.co
Developers download quantized large language models (LLMs) from public registries to run them locally. None of the registries we examined documented functional testing of their files. We tested all 305 eligible code-capable GGUF tags (the llama.
By Aditi Patodiya · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.