Learning Length-Extrapolatable Recurrent Models
Researchers propose Credit Stabilization through Time, a training method letting recurrent models extrapolate up to 128x their training length.
The paper argues that length extrapolation failure in BPTT-trained recurrent models is better explained through state credit, the signal through which future losses reach earlier recurrent states. It introduces Credit Stabilization through Time (CST), which locally rescales the state-credit signal during backpropagation without rotating the corrected component or changing forward computation. Controlled experiments show improved performance beyond the training horizon, with gains at up to 128x the training length.
25