How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
Analysis of DeepSeek-V4-Flash shows four-stream mHC residual blocks use only about two streams effectively, with late-layer mixing providing little benefit.
1 stories, newest first · one card per story
Analysis of DeepSeek-V4-Flash shows four-stream mHC residual blocks use only about two streams effectively, with late-layer mixing providing little benefit.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views