| title | Rethinking Sequence Compression with Test-time Regression |
|---|---|
| authors | Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu |
| date | 2026-10-22 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.31574 |
| affiliation | CUHK SphereLab |
| upvotes | 21 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models. |
cite
@misc{yang2026rethinking,
title={Rethinking Sequence Compression with Test-time Regression},
author={Sihan Yang and Jie Huang and Hanxuan Li and Yifan Guo and Weigao Sun and Yiran Zhong and Weiyang Liu},
year={2026},
eprint={2610.31574},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.31574}
}