How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Reproduction study finds Orthrus speculative-decoding trajectories match the reference model in only ~45% of cases under BF16, but 100% under FP32.
Researchers independently reproduced Orthrus, a hybrid autoregressive-diffusion architecture claiming lossless speculative decoding via intra-model consensus, testing exact trajectory matching on 1,190 prompts across 12 domains. Under BF16, exact matching occurred in only 45% of cases for the authors' checkpoint and 43% for an independently trained model, with matching probability strongly tied to reference-model response-conditional perplexity. Despite trajectory divergence, downstream lm-eval-harness benchmarks showed no systematic degradation, while FP32 evaluation yielded exact matching on all prompts.
- BF16 exact trajectory matching only 43-45% across 1,190 prompts
- FP32 inference yields exact matching on all evaluated prompts
- No systematic downstream benchmark degradation despite divergence
- Trajectory equivalence should be evaluated separately from task performance
Full article159 words · extracted from huggingface.co · click to collapse
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model. We independently reproduce Orthrus and examine this claim under different numerical precisions. Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors' checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains. The probability of exact matching is also strongly associated with the response-conditional perplexity of the reference model. Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. In contrast, repeating the trajectory evaluation with FP32 yields exact trajectory matching on all evaluated prompts. These results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.15504