RangeSWA provides training-free length extrapolation for diffusion language models
Original title: RangeSWA: Training-Free Length Extrapolation for Continuous Diffusion Language Models
RangeSWA reduces perplexity and repetition in long-text generation using sliding-window attention and positional remapping.
Continuous diffusion language models suffer from text repetition, semantic drift, and quality degradation beyond training lengths. RangeSWA addresses these via local sliding-window attention, remote block summaries, dynamic retrieval, positional remapping, conditioning attention correction, and adaptive window scheduling. Experiments across models confirm lower perplexity and fewer repetitions in extended outputs.
- Sliding-window attention preserves local details
- Remote summaries enable long-range context
- Positional remapping mitigates degradation
- Dynamic retrieval supports extended generation
- Tests show reduced perplexity rates
- AI models
- Continuous diffusion language models
From the original article
1 min read at arxiv.orgContinuous diffusion language models show promising potential in text generation tasks, but are prone to problems such as text repetition, semantic drift, and generation quality degradation when the generation length exceeds the training length. To address these challenges, we propose RangeSWA, a training-free length extrapolation method.
By Guangfu Guo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.