How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Reproduction study finds Orthrus speculative-decoding trajectories match the reference model in only ~45% of cases under BF16, but 100% under FP32.
Researchers independently reproduced Orthrus, a hybrid autoregressive-diffusion architecture claiming lossless speculative decoding via intra-model consensus, testing exact trajectory matching on 1,190 prompts across 12 domains. Under BF16, exact matching occurred in only 45% of cases for the authors' checkpoint and 43% for an independently trained model, with matching probability strongly tied to reference-model response-conditional perplexity. Despite trajectory divergence, downstream lm-eval-harness benchmarks showed no systematic degradation, while FP32 evaluation yielded exact matching on all prompts.
- BF16 exact trajectory matching only 43-45% across 1,190 prompts
- FP32 inference yields exact matching on all evaluated prompts
- No systematic downstream benchmark degradation despite divergence
- Trajectory equivalence should be evaluated separately from task performance
From the original article
1 min read at huggingface.co
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model.
By Ilya Koziev, Leonid Sinev, Ivan Oseledets · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.