LOWN skips image tokens in diffusion VLMs when predictions agree
Original title: Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs
Without training, LOWN keeps 99.6% of LLaDA-V's score at 35.3% of average decoding FLOPs.
Diffusion vision-language models process the full input at every denoising step, and visual tokens are 89-97% of that input on the studied benchmarks. The authors find removing the image changes an unmasked token at 22.8% of steps on average, and as few as 7.9%. LOWN predicts without the image and reuses the last image-conditioned result when the two agree, also setting how many tokens to unmask. It needs no training or tuned threshold and composes with visual token pruning and key-value caching.
- Removing the image changes an unmasked token at only 22.8% of steps on average.
- The smallest measured image-sensitive share of steps is 7.9%.
- LOWN retains 99.6% of LLaDA-V's score across ten benchmarks.
- Average decoding cost falls to 35.3% of baseline FLOPs.
- It generalizes across LLaDA-based diffusion vision-language models.
From the original article
1 min read at huggingface.co
Diffusion vision-language models (DVLMs) process their entire input at every denoising step. Since visual tokens make up 89-97% of this input on the benchmarks we study, most of the decoding computation is spent repeatedly processing the same image.
By Donghyeon Cho, Joowon Kim, Seungho Shin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.