Hierarchical Residuals Decouple Residual and Compute Width
Original title: Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width
Block-wise grouping with wider residuals and learned projections improves FLOPs efficiency while preserving standard layer structure.
Residual connections in Transformers tie residual width to per-layer compute width. Hierarchical Residuals (HiRes) groups layers into blocks using standard d-dimensional PreNorm layers inside blocks and wider D-dimensional PostNorm outer residuals between blocks, bridged by learned read/write projections. Across model sizes and depths, HiRes outperforms vanilla residuals with 1.30-fold FLOPs efficiency gains on OLMoE-1B-7B pretraining for 500B tokens, achieving 0.0208 lower loss and 1.14 higher downstream accuracy at lower activation memory and I/O costs than alternatives.
- Decouples residual width from compute width
- Learned projections bridge block spaces
- 1.30-fold FLOPs efficiency gain
- 0.0208 lower training loss on OLMoE-1B-7B
- Competitive with VWN and AttnRes at lower memory
From the original article
1 min read at huggingface.co
Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed.
By Ziteng Wang, Jianqiao Lu, Xinyu Ma · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.