VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs
VisionWeave cuts MLLM visual tokens 43% on Qwen3.8-27B while retaining 98.9% performance and 2.3x SGLang throughput.
VisionWeave teaches multimodal models to allocate elastic visual tokens by content rather than using fixed dense patches. A gated spatial pooler and granularity router are trained by self-distillation on Qwen3.5-4B and scaled to Qwen3.8-27B with over 30K A100 GPU-hours. On Qwen3.8-27B it saves 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, beating fixed 50% pruning that kept only 88%. On SGLang it delivers 2.3x throughput, 54.4% lower mean TTFT, and 60.6% lower mean TPOT.
- A gated pooler and granularity router allocate fine and coarse visual tokens.
- Trained by self-distillation on Qwen3.5-4B and scaled to Qwen3.8-27B.
- Saves 43.0% tokens while keeping 98.9% of native performance across eight benchmarks.
- SGLang deployment yields 2.3x throughput and cuts mean TTFT by 54.4%.
Full article245 words · extracted from huggingface.co · click to collapse
Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.07987