FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
FreeSolo adds linear-time sparse attention that extrapolates context length without long-context fine-tuning.
FreeSolo is a native sparse attention design with a short-range sliding window and a long-range branch that attends directly to selected keypoint cells. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection using short continued pretraining, while FS-B searches balanced trees at O(N log^2 N) including radix-sort construction. The method shows stronger zero-shot length extrapolation than baselines without long-context fine-tuning, avoiding both fixed recurrent-state limits and quadratic full attention.
- FreeSolo pairs a sliding-window branch with long-range keypoint-cell attention.
- FS-L learns prefix pruning via short continued pretraining for linear time.
- FS-B searches balanced trees at O(N log squared N), including radix-sort construction.
- It extrapolates beyond trained length better than baselines without long-context fine-tuning.
From the original article
1 min read at huggingface.co
Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length.
By Zekun Shi, Alastair James Arthur Price · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.