TempCloze: Can Video-LLMs Identify the Missing Middle?
TempCloze benchmark tests Video-LLMs' temporal reasoning with 1,521 videos, finding temporal alignment is the primary failure mode across 31 models.
TempCloze is a video cloze benchmark in which models must identify the true missing middle clip given the beginning and ending clips, using 1,521 carefully filtered videos from seven sources, mostly long-take and egocentric footage. Distractors are constructed along three dimensions: Semantic, Alignment and Progression, with shared scenes and objects to reduce appearance cues. Evaluation of 10 proprietary and 21 open-source Video-LLMs found Alignment is the primary bottleneck, with models often recognizing plausible semantics and local event progression but struggling with temporal alignment.
- Benchmark uses 1,521 filtered videos, mostly long-take and egocentric, from seven sources.
- Models pick the true missing middle from four candidates built along Semantic, Alignment and Progression dimensions.
- Evaluates 10 proprietary and 21 open-source Video-LLMs on TempCloze-Mixed and TempCloze-Hard variants.
- Alignment proves the primary bottleneck despite models recognizing plausible semantics and local event progression.
From the original article
1 min read at huggingface.co
Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correlations, or language priors. To reduce such shortcuts, we introduce TempCloze, a video cloze benchmark for evaluating visual temporal reasoning in Video-LLMs.
By Wenqi Pei, Henry Hengyuan Zhao, Yilai Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.