Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding
Entropy and CKA layer selection improves EAGLE3-style speculative decoding over the fixed [2, N//2, N-3] baseline.
The paper analyzes LLM decoder hidden states for speculative decoding using matrix-entropy informativeness and centered kernel alignment diversity. Token-entropy filtering drops low predictive-entropy tokens before layer statistics are computed. Entropy- and CKA-based selection beat fixed baselines, including the default [2, N//2, N-3] rule, in EAGLE3-style speculative decoding. The work also maps geometric and informational compartments across decoder depth.
- Layer choice uses matrix entropy for informativeness and CKA for diversity.
- Token-entropy filtering removes low-entropy tokens from selection statistics.
- Selected layers beat the default [2, N//2, N-3] EAGLE3 configuration.
- Analysis shows distinct informational compartments across decoder depth.
From the original article
1 min read at huggingface.co
Decoder hidden states from different layers of large language models (LLMs) often encode complementary information. However, existing speculative decoding methods typically rely on a small set of target-model layers chosen using fixed, position-based heuristics. Selecting an optimal subset of layers is inherently a combinatorial problem, making exhaustive evaluation infeasible even for moderately deep models.
By Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.