TTR-Compressor uses test-time regression for sequence compression
Original title: Rethinking Sequence Compression with Test-time Regression
On visual token merging it matches an MLP merger while using 55.6% fewer visual tokens.
The paper treats sequence compression as a regression problem solved in closed form during the forward pass and learned end to end. TTR-Compressor is proposed as an alternative to pooling and learned projections. Used for block-wise sparse attention, it improves RULER scores over attention-based compressors by a large margin. For multimodal models, it beats a standard MLP visual-token merger and matches that merger with 55.6% fewer visual tokens.
- Input tokens define a regression whose coefficients are the compressed representation.
- Coefficients are computed in closed form during the forward pass.
- The regression problem is trained with backpropagation.
- Evaluations cover sparse attention and visual token merging.
From the original article
1 min read at huggingface.co
Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation.
By Sihan Yang, Jie Huang, Hanxuan Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.