Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Models
CIRA induces LVLM failures only after visual-token compression, with a 20.35% mean failure rate.
The paper defines compression-specific failures in large vision-language models: adversarial inputs that stay correct under full-token inference but fail after visual-token compression. Their CIRA attack, using only a white-box vision encoder, achieves a mean compression-specific failure rate of 20.35% across 12 dataset-compressor settings at four budgets, while full-token attack success is 6.92%. A cross-view selection-stabilization defense substantially reduces the attack, though an adaptive variant partially restores it.
- A compression-specific failure is correct at full tokens but wrong after compression.
- CIRA needs white-box vision-encoder access, not the language model or compressor.
- Mean compression-specific failure rate is 20.35% across 12 settings.
- Full-token attack success is 6.92%; a stabilization defense largely suppresses CIRA.
From the original article
1 min read at arxiv.orgVisual token compression reduces the inference cost of Large Vision-Language Models (LVLMs). However, aggregate robustness measures do not reveal whether a particular adversarial failure is induced by compression or inherited from the underlying model.
By Qiankun Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.