How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
Analysis of DeepSeek-V4-Flash shows four-stream mHC residual blocks use only about two streams effectively, with late-layer mixing providing little benefit.
The study examines the four-stream residual pathway of DeepSeek-V4-Flash, finding typical attention or FFN sites effectively use about two streams and that residual mixing is modest, occurring primarily in early layers. Replacing late mixers with identity increases C4 perplexity by only 1.9% while replacing early mixers raises it by 41%. Retaining the three largest routing weights per token increases perplexity by at most 2.7%, showing the model uses only part of the flexibility afforded by the four-stream design.
- Typical blocks use ~2 of 4 streams; dominant stream shifts across depth
- Late-layer mixers replaceable by identity with only 1.9% perplexity increase
- Early mixer structure matters more than token-wise variation
- Routing weights can be pruned to top-3 per token with minimal impact
- Vendors
- DeepSeek
- AI models
- DeepSeek-V4-Flash
From the original article
1 min read at arxiv.orgHyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations.
By Pengxiang Zhao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.