Test-time Regression Improves Sequence Compression
Original title: Rethinking Sequence Compression with Test-time Regression
TTR-Compressor uses end-to-end learned regression for sequence compression, outperforming pooling and projections in language and multimodal models.
Sequence compression is critical for efficient language and multimodal models, yet pooling or projections underperform. TTR-Compressor frames input tokens as a regression problem learned end-to-end via backpropagation, with coefficients computed in closed form during forward pass. Evaluated as compressor for block-wise sparse attention and visual token merger in multimodal LLMs, it improves RULER performance over attention-based compressors and matches MLP merger performance with 55.6% fewer visual tokens.
Unconfirmed: Evaluations limited to tested models and tasks
- End-to-end backpropagation learned regression
- Closed-form coefficients in forward pass
- 55.6% fewer visual tokens for same performance
- Improves RULER over attention-based compressors
- Competitive on GSM8K MATH-500 HumanEval
From the original article
1 min read at arxiv.orgSequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation.
By Sihan Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.