How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
Analysis of DeepSeek-V4-Flash shows four-stream mHC residual blocks use only about two streams effectively, with late-layer mixing providing little benefit.
The study examines the four-stream residual pathway of DeepSeek-V4-Flash, finding typical attention or FFN sites effectively use about two streams and that residual mixing is modest, occurring primarily in early layers. Replacing late mixers with identity increases C4 perplexity by only 1.9% while replacing early mixers raises it by 41%. Retaining the three largest routing weights per token increases perplexity by at most 2.7%, showing the model uses only part of the flexibility afforded by the four-stream design.