Learning Length-Extrapolatable Recurrent Models
Researchers propose Credit Stabilization through Time, a training method letting recurrent models extrapolate up to 128x their training length.
The paper argues that length extrapolation failure in BPTT-trained recurrent models is better explained through state credit, the signal through which future losses reach earlier recurrent states. It introduces Credit Stabilization through Time (CST), which locally rescales the state-credit signal during backpropagation without rotating the corrected component or changing forward computation. Controlled experiments show improved performance beyond the training horizon, with gains at up to 128x the training length.
- Reframes length extrapolation failure around state credit rather than vanishing or exploding gradients.
- CST rescales the state-credit signal during backprop while leaving forward computation unchanged.
- Performance gains beyond the training horizon observed at up to 128x the training length.
Full article154 words · extracted from arxiv.org · click to collapse
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.09157