Fractional State Space Transition for Long Sequence Modeling
FRAC, a selective state-space model using fractional dynamics, improves long-context memory over exponential-forgetting SSM baselines.
Researchers introduce FRAC, a selective state-space architecture derived from fractional dynamics that replaces exponential forgetting with power-law long memory. It approximates a heavy-tailed kernel with a finite, log-spaced sum of exponential modes, supporting parallel training and prefill while keeping bounded-state decoding. Experiments, including 1.3-billion-parameter language modeling, show consistent long-context gains over state-of-the-art SSM baselines while remaining competitive on short context.
- FRAC replaces exponential decay with power-law long memory.
- A log-spaced sum of exponentials keeps the recurrent state bounded.
- The module supports parallel training and autoregressive decoding.
- A 1.3B language model improves long-context performance.
From the original article
1 min read at huggingface.co
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges.
By Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.