VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification
VDiff-Bench, a 1,756-question benchmark, shows multimodal LLMs struggle with fine-grained image-difference identification, scoring as low as 8.7% on low-level changes.
VDiff-Bench is a multiple-choice benchmark of 1,756 four-way questions over image pairs covering 10 change categories including position, motion, color, texture, OCR/text and illumination, with curated hard negatives. Evaluation of 11 state-of-the-art open- and closed-source MLLMs shows fine-grained visual comparison remains brittle: 7-8B-scale open-source models score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture. Notably, Grok 4.3 shows a sharp performance drop on noise and texture differences, falling behind large open-source models like Kimi K2.5 and K3.
- Benchmark uses ground-truth-conditioned negatives requiring models to distinguish actual changes
- Three 7-8B open-source MLLMs score 8.7-33.3% on low-level changes
- Grok 4.3 underperforms on noise/texture differences despite strong commercial peers
- Positions VDiff-Bench as a diagnostic beyond single-image vision-language tasks
- AI models
- VDiff-BenchGrok 4.3Kimi K2.5Kimi K3
From the original article
1 min read at huggingface.co
Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification.
By Yixin Wan, Tianle Zheng, Kai-Wei Chang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.