SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
SpectralShift extends Gated DeltaNet context windows via spectral reparameterization of alpha projections, improving long-context capabilities without architectural changes.
The paper studies long-context extension of Gated DeltaNet (GDN) linear attention from the spectral perspective of its transition matrix, identifying two key factors: a broad slow spectral band aligned with target dependency lengths, and preserved fast-decaying modes for state clearing and context switching. SpectralShift reparameterizes the alpha projections initialization to reshape the decay spectrum and adds a learning-rate scaling for alpha projections during long-context continual pretraining. Experiments show consistent long-context improvements over direct continued pretraining, and the code is open-sourced.