LooM Enables Loop-Invariant Memory for Causal Encoder-Decoder Models
LooM loops only decoder layers after the first in causal encoder-decoder models to provide budgets from 3 to 21 executions per token without growing cache size.
Stories per day · 30 days
2 RSS
Relatedalignment1astra1bptt1chain-of-thought1kv-cache-quantization1llm-architecture1llm-inference1memory1memory-optimization1model-release1openai1safety1scaling-laws1training-compute1transformer-architecture1transformers1
20 stories, newest first · one card per story
Stories tagged #looped-transformers
LoopVQ enables sub 1-bit KV cache quantization for looped transformers
LoopVQ quantizes KV cache at 0.5 bits per element for looped transformers, nearly halving decode latency at 32K context.
Scaling law favors full BPTT over truncated looped transformers
Two fully trained loops beat the compute-optimal non-looped model with about 27% fewer non-embedding parameters.
OpenAI will reportedly ship at least one model this year but its flagship still needs alignment work
OpenAI will reportedly ship one or two models by year-end, but its looped-transformer flagship still needs alignment work and safety test infrastructure.
Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
InfiLoop adds loop-native residuals so looped Transformers keep improving past 20,000 test-time steps.
SanSi: A Looped Typed Decision Model for System 1.5 Thinking
SanSi adds looped System 1.5 thinking to typed decision models, reaching 72% accuracy.
LiFT: Loop Flow Transformers
LiFT loops a shared Diffusion Transformer and lowers ImageNet FID while cutting parameters and FLOPs.
Decoding Looped Transformers Better for (Almost) Free
Training-free LoopCD improves looped Transformer decoding and can cut recurrent compute by up to 48%.
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
LOOM scales looped Mixture-of-Experts Transformers to 9-12 loops, cutting 1.7B model perplexity from 9.62 to 7.77 with accuracy gains.
Decoding Looped Transformers Better for (Almost) Free
LoopCD uses earlier recurrent states for contrastive decoding, boosting looped Transformers while cutting inference FLOPs.
Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model looped recurrence and MoE sparsity, matching larger models at equal compute.
Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model recurrence and sparsity, predicting looped MoE loss and guiding compute-efficient design.
Scheduling Recursive Reasoning in Looped Transformers
TAPS adapts recurrent step size in looped transformers, raising accuracy and cutting loops without retraining.
How to Loop MoE: Flatten the Experts, Untie the Attention
Foil flattens experts and unties attention in looped MoE models, lowering pretraining loss at equal parameters and compute.
Improving Test-Time Scaling with Adaptive Looped Transformers
TaH2 adaptively loops tokens that benefit, steepening the AIME accuracy-compute slope 53 percent versus a non-looped baseline.
How to Loop MoE: Flatten the Experts, Untie the Attention
Foil flattens experts and unties attention so looped mixture-of-experts models train with lower loss.
Improving Test-Time Scaling with Adaptive Looped Transformers
TaH2 adaptively loops layers on hard tokens, steepening AIME test-time scaling by 53% over a non-looped baseline.
How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Researchers show model growth via looped transformers improves scaling exponents; a 7.4B architecture matches GPT-3 13B with roughly 20x less compute.
GPT-6 Astra, Looped Transformers, and Hidden Reasoning
OpenAI released GPT-6 Astra, its strongest model to date, with standout 3D rendering and computer-use performance and 99.9% on ARC-AGI-3.
4
[AINews] Muse Spark 1.3 matches GPT-5.6-Sol, confirming Meta Superintelligence as the newest Frontier Lab, >90% discount for training
Meta's Muse Spark 1.3 reportedly ranks as the world's #3 model, matching frontier models from OpenAI and Anthropic with planned open weights.
4