RangeSWA provides training-free length extrapolation for diffusion language models
RangeSWA reduces perplexity and repetition in long-text generation using sliding-window attention and positional remapping.
Stories per day · 30 days
3 stories, newest first · one card per story
Stories tagged #length-extrapolation
FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
FreeSolo adds linear-time sparse attention that extrapolates context length without long-context fine-tuning.
Learning Length-Extrapolatable Recurrent Models
Researchers propose Credit Stabilization through Time, a training method letting recurrent models extrapolate up to 128x their training length.