Looping cuts training FLOPs for uniform diffusion language models
Looped UDLMs match a roughly 0.5-trillion-token baseline with up to 2.8x fewer FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #looping
Looped uniform diffusion language models cut training FLOPs
Selective looping matches a non-looped UDLM's validation NLL with up to 3.3x fewer training FLOPs.