What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains
Researchers find signed full-precision LLMs cannot verify popular GGUF builds, with verifiability at 0 of 198 repos.
A cs.CR paper measures whether third parties can bit-for-bit re-derive signed LLMs after quantization. Across 198 in-scope top Hugging Face GGUF repositories, no artifact or model card records the build environment, so the Independent Verifiability Rate is 0/198. On four repos that disclose inputs, two omitted coordinates reproduced 7 of 8 Q4_K_M files on x86 and 5 of 8 on arm64, while the calibration imatrix had to be shipped and hash-bound. The authors say benign floating-point drift can be separated from uniform nibble-flip tampering, but a targeted embedding edit evades that statistic.
- None of 198 popular Hugging Face GGUF repos record the build environment.
- Independent Verifiability Rate is 0/198 under a recorded-inputs protocol.
- Two omitted build coordinates re-derived 7 of 8 files bit-for-bit on x86.
- One compiler setting gives a unique output at about +0.06% perplexity.
- Targeted embedding edits evade statistics, so exact re-derivation is required.
Full article277 words · extracted from arxiv.org · click to collapse
Model signing authenticates the full-precision weights W an LLM is released with, yet users run a quantized artifact Q whose producing transformation is not uniquely determined under current disclosure. We separate the non-environment inputs from the execution environment E and measure a quantity that, to our knowledge, has not been measured before: the Independent Verifiability Rate (IVR), the fraction of a popularity-sampled GGUF ecosystem that a third party can re-derive bit-for-bit. Across the 200 most-downloaded Hugging Face GGUF repositories (198 in scope), no artifact or model card records the build environment, so under a recorded-inputs protocol the IVR is 0/198. Yet the artifacts are not irreproducible. On four in-sample repositories that disclose their inputs, the card's information alone reproduces no Q4_K_M file, but two coordinates the card omits - the build's floating-point-contraction setting and, for hybrid models, the per-tensor quantization recipe - re-derive 7/8 target-quant files bit-for-bit on x86 (5/8 on arm64); the fp16 re-derives from the provider's checkpoint, but the calibration imatrix must be shipped and hash-bound, since recomputing it did not match. An older release additionally pins the instruction set and did not re-derive. Reproducing the divergence reported with llama.cpp PR #25353, we show that E is a necessary coordinate and that one compiler setting yields a unique output across the tested platforms at negligible quality cost (+0.06% perplexity). As a security characterization, benign floating-point non-determinism separates from uniformly placed nibble-flip tampering above a detection floor that grows with the environments a verifier must allow, but a targeted embedding edit evades the statistic, so the proposed protocol rests on exact re-derivation; adversarial-fp16 attacks lie outside it. We release the audit data and a re-runnable harness.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.19498