SphereLab · Technical Report

TTR-Compressor
Rethinking Sequence Compression
with Test-time Regression

Sihan Yang¹Jie Huang²Hanxuan Li¹Yifan Guo¹Weigao Sun³Yiran Zhong³Weiyang Liu^1,*

¹ The Chinese University of Hong Kong² Independent Researcher³ Token Foundry, Alibaba Group

* Corresponding author

Paper Code Model

The idea at a glanceOne operator. Two applications.

Comparison of pooling, fixed projection, and TTR joint regression, with applications to sparse attention and visual token merging. — **From aggregation to regression.** Input tokens define a regression problem; its fitted coefficients become the compressed representation.

+12.24 pts

RULER retrieval accuracy

vs. attention-weighted pooling

55.6%

Fewer visual tokens

54.85 vs. 54.93 average score

Learnable parameters

Base compressor · exact closed-form solve

Abstract

Sequence compression is a critical information bottleneck in efficient language and multimodal models. We revisit it through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. The regression objective is learned end-to-end, while its coefficients are computed efficiently in closed form during the forward pass.

The resulting TTR-Compressor fits tokens jointly through their second-order correlations. It improves retrieval in block-wise sparse attention and consistently outperforms the standard MLP visual merger, matching its performance with 55.6% fewer visual tokens.

01 / Method

A better summary starts
with the relationships.

Learn which regression problem to solve.
Adapt its solution to every input.

Construct from the input

Normalize tokens to form keys. A learned projection creates scalar targets from the original features, retaining their magnitudes.

Fit all tokens jointly

The Gram matrix captures inter-token correlations. Repeated evidence has diminishing influence, preserving distinctive information.

Solve exactly, efficiently

A small token-space Cholesky solve yields the representation. No iterative gradient updates are needed at inference.

The compression objective

h^{\star}=\underset{h}{\operatorname{arg\,min}}\;\left\{\frac{1}{2}\lVert Kh-v\rVert_{2}^{2}+\frac{1}{2}h^{\top}Dh\right\}

K=\operatorname{RowNorm}(X)

v=Xa

D=\operatorname{diag}\!\bigl(\operatorname{softplus}(\rho)\bigr)

The target projection $a$ and positive diagonal regularizer $D$ are learned during training and remain fixed at inference.

A small system, an exact solution.

When the compression window is smaller than the feature dimension ( $T\ll d$ ), the dual formulation replaces a $d\times d$ system with a $T\times T$ system.

Cholesky factorization avoids explicitly forming an inverse. The general cost is $\mathcal{O}(T^2d+T^3)$ , reducing to $\mathcal{O}(T^2d)$ in the targeted regime.

Dual formulation

\left(I_T+KD^{-1}K^{\top}\right)\alpha=v,

h^{\star}=D^{-1}K^{\top}\alpha.

T\times T

solve ·

2d

base parameters

02 / Why it works

Learn what matters.
Adapt to what repeats.

A task-supervised objective.
An input-dependent representation.

TTR-Compressor combines two complementary mechanisms: training learns which regression problem produces useful summaries, and the inner solve adapts each summary to the relationships among the current tokens. This connects the compression rule to both the downstream task and the structure of each input.

Bilevel learning diagram: task gradients learn targets and regularization, while an exact dual solve adapts to each input. Repeated tokens receive diminishing influence. — **Learning the objective, adapting the representation.** Task gradients pass through the exact solve. At inference, only the input-specific fitted coefficients change. Figure 2 from the paper.

01 / Task-supervised learning

The task shapes the regression.

Gradients from the downstream task pass through the exact solve to learn the target projection $a$ and diagonal regularizer $D$ . The objective is trained to produce useful representations, rather than merely to minimize the inner regression error.

The learned regularizer also shapes the similarity metric $KD^{-1}K^{\top}$ , determining which feature directions matter when comparing tokens. At inference, these learned parameters remain fixed.

02 / Second-order adaptation

Redundancy has diminishing influence.

The Gram matrix couples all tokens in a group. Their effective weights are solved jointly, so repeated evidence has diminishing rather than purely additive influence. Distinctive information can retain a larger relative contribution when supported by the regression targets.

In the figure’s example, three identical tokens and one orthogonal token have equal targets and $D=I$ . Pooling gives the two groups a 3 : 1 influence ratio; joint regression gives 3 : 2, increasing the distinct token’s relative share.

Why normalize only the keys?

Normalized keys make token similarity depend on direction, while targets retain the original feature magnitudes. If both used normalized features, the fixed projection $a$ would exactly fit the regression targets, weakening the need for input-dependent adaptation. This asymmetry helps the fitted representation respond to each sequence.

An intuition you can explore

More repetition.
Less domination.

Compare a group of identical tokens with one orthogonal, distinct token. Joint regression tempers the repeated group’s influence.

Repeated tokens 3

Illustrative setting: unit keys, equal targets, D = I. Bars show each group’s share of total coefficient mass; this is not a benchmark.

Repeated groupDistinct token

Additive pooling 3 : 1

Joint regression 3 : 2

The distinct token retains 40% of the total influence with TTR, compared with 25% under pooling.

03 / Results

More accurate retrieval.
Richer representations.

Faithful retrieval for language.
Compact representations for vision.

Block-wise sparse attention

Better summaries make better retrieval keys.

We vary only the compressor in the HiLS-Attention framework. With 345M-parameter models trained on approximately 30B tokens at an 8K context length, TTR improves average RULER retrieval accuracy across four tasks and context lengths from 4K to 32K.

RULER · Average accuracy (%) ↑

Retrieval that scales

51.44%

TTR with local convolution, compared with 39.20% for attention-weighted pooling.

At 1B parameters / 100B training tokens, the base TTR compressor reaches 79.96%, compared with 73.50% for attention-weighted pooling (Table 7).

View retrieval results by task and context length +

Visual token merging

Keep the understanding. Use fewer tokens.

Using the Qwen3-VL vision encoder and Qwen3-4B-Instruct language backbone, TTR improves the average score across ten benchmarks spanning visual question answering, knowledge, document understanding, and spatial reasoning.

Method	MMBench	MMStar	RWQA	MMMU	AI2D	OCRBench	InfoVQA	ChartQA	MMSI	ERQA	Avg.

Standard MLP · 2 × 254.93N / 4 visual tokens

TTR + Conv · 3 × 354.85N / 9 visual tokens

55.6%fewer visual tokens
at comparable average performance

Practical by design

Expressive compression.
Comparable efficiency.

The exact solve stays small. In the reported visual-merging setup, base TTR has training time comparable to MLP, with a slight reduction at 3 × 3 merging.

Merger	2 × 2	3 × 3
MLP	18.40 h	17.25 h
TTR-Compressor	18.55 h	17.02 h

04 / Citation

BibTeX

@article{yang2026ttrcompressor,
  title={Rethinking Sequence Compression with Test-time Regression},
  author={Yang, Sihan and Huang, Jie and Li, Hanxuan and Guo, Yifan
          and Sun, Weigao and Zhong, Yiran and Liu, Weiyang},
  journal={arXiv preprint arXiv:2610.31574},
  year={2026}
}

TTR-CompressorRethinking Sequence Compression with Test-time Regression

A better summary startswith the relationships.