How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Reproduction study finds Orthrus speculative-decoding trajectories match the reference model in only ~45% of cases under BF16, but 100% under FP32.
Researchers independently reproduced Orthrus, a hybrid autoregressive-diffusion architecture claiming lossless speculative decoding via intra-model consensus, testing exact trajectory matching on 1,190 prompts across 12 domains. Under BF16, exact matching occurred in only 45% of cases for the authors' checkpoint and 43% for an independently trained model, with matching probability strongly tied to reference-model response-conditional perplexity. Despite trajectory divergence, downstream lm-eval-harness benchmarks showed no systematic degradation, while FP32 evaluation yielded exact matching on all prompts.