DAGFormer plans Transformer cross-layer connections with a topology predictor
On matched OLMo-2 pairs, WikiText-2 perplexity falls 17% to 25% versus a vanilla Transformer.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #model-architecture
Extender adds a log channel to cut Transformer attention memory
At 924M parameters and extension size 32, authors report matching short-context accuracy and beating Transformers on long-context RULER tasks.