PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?
PlaylistEval tests video-language judges on 100-hour playlists, where frontier models reach only 75.4% accuracy.
PlaylistEval is an agentic framework for building video-language judge benchmarks over 100-hour playlist collections without human annotation. It generates 630 question-answer pairs across seven domains using causal degradation so differences require retrieval across the collection. On a stratified subset of 152 pairs, automatic labels agree with humans 93.0% of the time (IAA 0.781). Across 17 models from eight families, frontier judges reach only 75.4% pairwise accuracy, open-source judges trail, and accuracy drops as the playlist grows. The pipeline, benchmark, and evaluation code are released.
- PlaylistEval builds judge benchmarks over 100-hour playlists without human annotation.
- The set has 630 causally degraded answer pairs across seven domains.
- Human agreement is 93.0% on 152 pairs, with IAA 0.781.
- Frontier judges reach 75.4% pairwise accuracy; open-source judges lag further.
- Accuracy depends on multiple modalities and falls as the playlist grows.
- Products
- PlaylistEval
From the original article
1 min read at huggingface.co
Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether their judgments remain reliable when the evidence is buried in day-long videos has yet to be established. Existing benchmarks cannot answer this.
By Shayekh Bin Islam, Hwanjun Song · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.