AskStream benchmark tests when video models should ask
Original title: AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding
The strongest model scores only 52.18% Ask-F1 across 438 videos labeled for wait, ask, and answer.
AskStream is a multi-turn benchmark for streaming video understanding when information is incomplete because of the video or the user. It adds an ask action alongside wait and answer, covering four task categories and eight subtasks. The set has 438 videos and 2,714 manually annotated frame-level time windows of three types. In multiple-choice and open-ended settings, the authors report that even the strongest model reaches only 52.18% on Ask-F1.
Unconfirmed: The 52.18% Ask-F1 figure is the authors' result; the strongest model is not named.
- The benchmark includes four task categories and eight subtasks.
- Annotations cover 2,714 frame-level time windows.
- Window labels are wait, ask, and answer.
- Results cover both multiple-choice and open-ended evaluation.
From the original article
1 min read at huggingface.co
Recent advances in multimodal large language models have increased interest in interaction paradigms for video understanding. The evolution from offline to streaming and proactive interaction has gradually shifted models from passive responders toward more proactive agents that can decide when to answer.
By Lu Wang, Zhuoran Jin, Longxiang Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.