When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors
A new benchmark finds vision-language models drop spatial relations from the representations used downstream.
The paper introduces EICS, a metric for concept stability under controlled one-concept interventions, and CASP-VL, a benchmark of spatial perturbations derived from Visual Genome scene graphs. Across multiple vision-language architectures, spatial relations are encoded much less faithfully than lexical information despite above-chance compositional benchmark scores. The relation remains identifiable at predicate tokens but is lost in global representations formed by the EOS token or mean pooling.
- EICS checks whether one-concept interventions leave encoded concepts stable.
- CASP-VL derives controlled spatial perturbations from Visual Genome scene graphs.
- Spatial relations are represented less faithfully than lexical information.
- Predicate-token relation signals disappear after EOS or mean-pooled aggregation.
From the original article
1 min read at arxiv.orgVision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning.
By Shubh Garg · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.