Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models
Study explains why verbose prompts make vision-language models more robust to image corruption, framing cross-modal attention as a frequency filter.
The paper shows question wording affects vision-language model robustness in two opposite ways: verbose prompts increase robustness, while semantically complex or fine-grained questions increase fragility under image corruption. Both effects are explained by question-conditioned cross-modal attention inducing a spectral filter over image patches, with answer drift largest when the filter and corruption share spatial frequencies. Experiments on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR show verbose paraphrasing reduces drift variance by 70-81% on 8B models and improves accuracy.