Skip to content
NewA calmer, faster ZeroHour

Quantization moves more Clef decisions than engine or restart drift

Original title: The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models

Hugging Face daily papersby Aleks Blagojevich, Trevor Sunderland ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

On 2,400 decisions, every 4-bit interval excluded non-quantization sources, while accuracy hid most of the decision changes.

Researchers compared quantization with serving changes on Cloudflare's Clef-flash (9B) and Clef (27B) using 2,400 public decisions on an NVIDIA GB10. Sources that are not quantization moved 0 to 0.92% of decisions. Inside one engine, 4-bit formats moved more decisions than restarts, engine changes, batch-size changes or an unasked kernel on both models. Over 31,275 questions, Clef's Q4_K_M build changed accuracy by a net six questions while an exploratory check found 436 decisions worse.

Unconfirmed: A third Clef instance comparison and the 436 worse decisions are described as exploratory.

Key facts
  • llama.cpp restarts were bit-identical on every record for both models.
  • vLLM restarts flipped 0, 5 and 20 Clef-flash decisions at BF16, FP8 and NVFP4.
  • Q4_K_M moved 5.79% of Clef-flash decisions and 3.50% of Clef decisions.
  • NVFP4 cost Clef-flash 0.95 accuracy points and Clef 0.65 points.
  • Two of five pre-registered predictions failed, and the same two failed on the 27B rerun.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

A quantized model is usually judged by comparing it with its BF16 original, but that comparison changes more than the weights. It often changes the serving engine, it always runs on a different server instance, it may change how requests are batched, and on some stacks the library selects a different kernel depending only on which packages import.

Hugging Face daily papers

By Aleks Blagojevich, Trevor Sunderland · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.