Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
delta-Vision preserves every visual token and replaces repeated Transformer updates with lightweight MLP memory adapters.
Long visual token sequences dominate compute in multimodal LLMs, and token pruning discards evidence later layers may need. The authors show that after blocking visual-to-text attention, restoring a few directions recovers most accuracy, and lightweight MLPs predict layer-specific visual states with high cosine similarity. delta-Vision replaces repeated Transformer updates of visual tokens with low-rank adapters that build layer-wise visual memories while keeping every token available for text retrieval. It reports higher accuracy than pruning baselines on image and video benchmarks at comparable or lower compute.
- Pruning visual tokens permanently drops evidence later layers may need.
- Visual influence on text sits in a low-dimensional subspace.
- Lightweight MLPs reconstruct layer-wise visual states with high cosine similarity.
- delta-Vision beats pruning baselines on image and video benchmarks at similar cost.
Full article186 words · extracted from huggingface.co · click to collapse
Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models~(MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose δ-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks, δ-Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.34972