LooM Enables Loop-Invariant Memory for Causal Encoder-Decoder Models
LooM loops only decoder layers after the first in causal encoder-decoder models to provide budgets from 3 to 21 executions per token without growing cache size.
Stories per day · 30 days
Relatedlong-context1looped-transformers1memory1memory-initialization1mirrormoe1mixture-of-experts1routing1scaling1
3 stories, newest first · one card per story
Stories tagged #llm-architecture
DAMI improves long-context retrieval in gated linear attention models
DAMI jointly configures memory parameters for better retrieval accuracy across multiple architectures.
Rethinking Routing Exploration in Modern Mixture-of-Experts
MirrorMoE fixes MoE routing under-exploration from shared experts by suppressing unneeded experts per token.