Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models
Study explains why verbose prompts make vision-language models more robust to image corruption, framing cross-modal attention as a frequency filter.
The paper shows question wording affects vision-language model robustness in two opposite ways: verbose prompts increase robustness, while semantically complex or fine-grained questions increase fragility under image corruption. Both effects are explained by question-conditioned cross-modal attention inducing a spectral filter over image patches, with answer drift largest when the filter and corruption share spatial frequencies. Experiments on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR show verbose paraphrasing reduces drift variance by 70-81% on 8B models and improves accuracy.
- Verbose questions broaden the attention frequency filter, improving robustness
- Fine-grained questions concentrate the filter, increasing corruption sensitivity
- Verbose paraphrasing cut drift variance 70-81% on 8B models
- Tested on Qwen3-VL and LLaVA-OneVision over GQA and CLEVR
Full article160 words · extracted from arxiv.org · click to collapse
Vision-language models (VLMs) are fragile under image corruption. We find that the wording of the question affects VLMs in two opposite ways. Verbose questions make VLMs substantially more robust---e.g., rephrasing "Is there a cat?" into "Please look carefully and answer: is there a cat?". Conversely, VLMs become more fragile under corruption when the question is semantically complex or finer-grained, e.g., "what colour is the cup left of the chair?" instead of "is there a cup?". Both effects stem from question-conditioned cross-modal attention, which induces a spectral filter over image patches: verbose questions broaden its frequency support, while fine-grained questions concentrate it onto fewer visual scales. The model's answer drifts most when this filter and the corruption sit on the same spatial frequencies. We test the filter view on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR; verbose paraphrasing reduces drift variance by 70--81% on the 8B models. The practical recipe---pad the prompt---further yields measurable gains in accuracy, even under image corruption.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20139