DAGFormer plans Transformer cross-layer connections with a topology predictor
Original title: DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections
On matched OLMo-2 pairs, WikiText-2 perplexity falls 17% to 25% versus a vanilla Transformer.
DAGFormer is a Transformer whose cross-layer connections are planned by a jointly trained topology predictor. Beyond residual links, it adds direct earlier-to-later connections and sets every connection strength from the input tokens, planning the whole model at once rather than layer by layer. On matched OLMo-2 pairs from 75M to 1B parameters, WikiText-2 perplexity drops 17% to 25% versus a vanilla Transformer, and a 1B model trained on 5B tokens beats a vanilla model trained on 10B. It also leads four routing baselines at 75M, 300M, and 1B, and gains carry to 13 downstream tasks, instruction finetuning, and pruning.
- Mean accuracy rises 2.0 to 2.5 points at 300M and 1B on 13 tasks.
- AlpacaEval win rate against the vanilla Transformer is 63% to 70%.
- Under structured pruning it degrades least in five of six settings.
From the original article
1 min read at huggingface.co
We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor. Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens.
By Xiaocong Yang, Yuren Hao, ChengXiang Zhai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.