Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Probing study shows vision encoders make canonical color linearly decodable from grayscale images and tie it to object identity.
Researchers use canonical color as a controlled testbed for measuring conceptual (not just visible) information in vision encoder representations. A dataset of objects with canonical colors was built, and probes on both color and grayscale images show canonical color remains decodable even when color is removed from the input, linked to predicted object identity. Extending to full VLMs, they find post-training has a surprisingly large effect on color decodability in the vision encoder.
- Canonical color remains linearly decodable from grayscale inputs
- Decodable color is tied to predicted object identity, indicating conceptual linkage
- VLM post-training substantially changes color decodability in the vision encoder
- Framework offers a controllable lens for tracing object-level semantics in multimodal models
From the original article
1 min read at arxiv.orgVisual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image.
By Xiaofu Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.