Scaling law favors full BPTT over truncated looped transformers
Two fully trained loops beat the compute-optimal non-looped model with about 27% fewer non-embedding parameters.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #bptt
Learning Length-Extrapolatable Recurrent Models
Researchers propose Credit Stabilization through Time, a training method letting recurrent models extrapolate up to 128x their training length.