PAR aligns looped model steps for longer state tracking
On swap-S5, a Qwen2.5-0.5B model trained on at most 32 moves reaches 95.5% full-state accuracy at 512 moves.
Stories per day · 30 days
Relatedlanguage-models3reasoning3architecture2hybrid-architectures2looped-transformers2mixture-of-experts2scaling-laws2softmax-attention2sparsity2state-tracking2algorithmic-generalization1inference-scaling1length-generalization1looped-language-models1looped-models1pretraining1qwen1retrieval1
7 stories, newest first · one card per story
Stories tagged #recurrence
Competing learning found in hybrid recurrence-attention models
Stopping recurrence updates mid-training lets attention learn retrieval; staged pretraining improved models up to 1B parameters.
Hybrid models suppress attention retrieval unless recurrence updates pause
A staged pretraining procedure that freezes recurrence mid-training improves loss, reasoning, and retrieval on models up to 1B parameters.
Recurrent Looped Transformer
Recurrent Looped Transformers track algorithmic state far beyond training length via per-token feedback.
Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model looped recurrence and MoE sparsity, matching larger models at equal compute.
Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model recurrence and sparsity, predicting looped MoE loss and guiding compute-efficient design.
What Makes Recurrence Effective in Looped Language Models?
Looped language models gain reasoning from recurrence beyond training, but extra loops can hurt knowledge.