Quantization moves more Clef decisions than engine or restart drift
Original title: The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models
On 2,400 decisions, every 4-bit interval excluded non-quantization sources, while accuracy hid most of the decision changes.
Researchers compared quantization with serving changes on Cloudflare's Clef-flash (9B) and Clef (27B) using 2,400 public decisions on an NVIDIA GB10. Sources that are not quantization moved 0 to 0.92% of decisions. Inside one engine, 4-bit formats moved more decisions than restarts, engine changes, batch-size changes or an unasked kernel on both models. Over 31,275 questions, Clef's Q4_K_M build changed accuracy by a net six questions while an exploratory check found 436 decisions worse.
Unconfirmed: A third Clef instance comparison and the 436 worse decisions are described as exploratory.
- llama.cpp restarts were bit-identical on every record for both models.
- vLLM restarts flipped 0, 5 and 20 Clef-flash decisions at BF16, FP8 and NVFP4.
- Q4_K_M moved 5.79% of Clef-flash decisions and 3.50% of Clef decisions.
- NVFP4 cost Clef-flash 0.95 accuracy points and Clef 0.65 points.
- Two of five pre-registered predictions failed, and the same two failed on the 27B rerun.
- Vendors
- NVIDIA
- Products
- vLLMllama.cppNVIDIA GB10
- Organizations
- Cloudflare
- AI models
- Clef-flashClef
From the original article
1 min read at huggingface.coA quantized model is usually judged by comparing it with its BF16 original, but that comparison changes more than the weights. It often changes the serving engine, it always runs on a different server instance, it may change how requests are batched, and on some stacks the library selects a different kernel depending only on which packages import.
By Aleks Blagojevich, Trevor Sunderland · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.