SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
SpectralShift extends Gated DeltaNet context windows via spectral reparameterization of alpha projections, improving long-context capabilities without architectural changes.
The paper studies long-context extension of Gated DeltaNet (GDN) linear attention from the spectral perspective of its transition matrix, identifying two key factors: a broad slow spectral band aligned with target dependency lengths, and preserved fast-decaying modes for state clearing and context switching. SpectralShift reparameterizes the alpha projections initialization to reshape the decay spectrum and adds a learning-rate scaling for alpha projections during long-context continual pretraining. Experiments show consistent long-context improvements over direct continued pretraining, and the code is open-sourced.
- Identifies slow spectral band breadth and fast-decaying modes as governing long-range retrieval.
- Reparameterizes alpha projection initialization to enhance slow propagation capacity.
- Adds learning-rate scaling for alpha projections to facilitate long-context training.
- Code open-sourced at github.com/RUCAIBox/GDN-SpectralShift.
- Tags
- continual-pretraininggated-deltanetlinear-attentionlong-contextresearchspectral-reparameterization
- Products
- Gated DeltaNetSpectralShift
From the original article
1 min read at huggingface.co
Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linear attention state dynamics.
By Zian Liu, Yiwen Hu, Zican Dong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.