Huginn-style looped models beat Ouro at matched scale
Original title: Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
A Huginn-style MoE uses 75% fewer resident parameters than a 112-layer feedforward baseline under matched FLOPs.
The paper isolates recurrence placement, input injection, and recurrent-state design in looped language models trained from scratch at matched scale. A prelude-recurrent block-coda sandwich improves context extraction and reasoning, while input injection mainly helps knowledge retrieval and context use. At 730M parameters and 336B tokens, the complete Huginn architecture outperforms Ouro on all ten benchmarks, and the advantage transfers to MoE models. Under matched training and inference FLOPs, Huginn uses 75% fewer resident parameters than a 112-layer feedforward MoE baseline.
- Random initialization and separate H/L recurrent states give mixed or negative returns.
- At 8BA0.8B trained on 500B tokens, Huginn beats Ouro on nearly all benchmarks.
- Huginn also shows more balanced expert routing than Ouro.
- It beats the feedforward baseline on DROP and GSM8K and matches MATH500.
From the original article
1 min read at huggingface.co
Looped language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization.
By Benhao Huang, Chufan Shi, Junlin Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.