DEPICT: Scoring Text-to-Image Alignment by Answer Agreement
DEPICT scores text-to-image alignment by answer agreement, raising negation accuracy from 19% to 88%.
DEPICT is a training-free text-to-image alignment metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighted by how decisively the caption determines each question. Combining that agreement score with a holistic score recovers context lost during question decomposition. The agreement rule raises negation accuracy from 19% to 88%. On five benchmarks and eleven backbones from three model families, DEPICT surpasses all training-free metrics and exceeds fine-tuned evaluators on two of three human-correlation benchmarks.
- Training-free metric uses answer agreement instead of fixed YES references
- Negation accuracy rises from 19% to 88%
- Merges decomposed agreement with a holistic score
- Beats training-free metrics across five benchmarks and eleven backbones
- Exceeds fine-tuned evaluators on two of three human-correlation benchmarks
From the original article
1 min read at huggingface.co
Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations.
By Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.