It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
MIST shows irrelevant images change VLM judge labels about as often as aligned images, without improving agreement.
MIST pairs 200 ambiguous English sentences with an aligned image, a misleading image, or no image, while instructions require the label to come from the sentence alone. Across thirteen VLM judges, aligned images changed 20.5% of labels and misleading images 19.4%, both above the 11.6% shift from dropping the ignore-image instruction. Only 37% of labels that differed between the two images moved toward the depicted sense, and agreement with human annotators was unchanged.
- Aligned images flip 20.5% of labels; misleading images flip 19.4%.
- Only 37% of differing labels move toward the depicted sense.
- Human agreement is unchanged across absent, aligned, and misleading images.
- The effect appears in all thirteen judges, including alt-test passers.
From the original article
1 min read at huggingface.co
Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions.
By Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.