Sable proves bit-exact quantized LLM inference on one GPU
Original title: Sable: A Small-Field Sumcheck Argument for Bit-Exact Inference of Large Language Models
Qwen2.5-32B takes 92.5 seconds for 4,096 tokens; proofs are not zero-knowledge.
Researchers present Sable, an argument system and single-GPU prover that a committed integer-quantized transformer produced a given token sequence bit for bit. It uses a fixed integer function with no floating-point step and a sumcheck-based constraint system over the 31-bit KoalaBear prime. On one NVIDIA RTX PRO 6000 Blackwell GPU it proves Qwen2.5-32B, Qwen3-32B and Qwen3-30B-A3B, at 41 to 50 times serving time. The interactive proof is not zero-knowledge and targets open-weight models.
- Qwen2.5-32B: 92.5 s for 4,096 tokens and 398 s for 16,384 tokens.
- Qwen3-32B and Qwen3-30B-A3B take 121.8 s and 84.7 s for 4,096 tokens.
- A 4,096-token proof is priced at 6.4 cents at public GPU rates.
- CPU verification takes 19 to 56 seconds; soundness error is 2^-103.
- Greedy tokens match BF16 at 86.3% of WikiText-2 positions.
- Vendors
- NVIDIA
- Products
- RTX PRO 6000 Blackwell
- AI models
- Qwen2.5-32BQwen3-32BQwen3-30B-A3B
From the original article
1 min read at huggingface.co
We present Sable, an argument system and single-GPU prover for the statement that a committed, integer-quantized transformer produced a given token sequence from a given prompt, bit for bit.
By Tomer Solberg · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.