Omni-InteractScope tests proactive help on continuous audiovisual streams
Original title: Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
Across 12 models, on-time first tool responses stay at or below 40% and reminder recall never exceeds 23%.
Omni-InteractScope is a benchmark of 1,479 human-annotated, cross-reviewed interactions and 1,790 temporal targets in eight task groups for omnimodal models watching continuous audiovisual streams. It covers tool invocation, query-free assistance, notifications, warnings, state tracking, and continuous responses. The authors report that all 12 evaluated models give an on-time first response for at most 40% of tool-invocation targets, with joint timing-and-tool accuracy at most 32.0%. Reminder recall never exceeds 23.0%, and only three models fully cover any of 26 guidance sequences.
Unconfirmed: Benchmark scores are the authors' experimental results on their own test set.
- The set has 1,479 interactions and 1,790 temporal targets.
- Joint timing-and-tool accuracy reaches at most 32.0%.
- Query-free reminder recall is lower than recall on explicit questions.
- Nine models cover none of the 26 guidance sequences in full.
- Tool-invocation scores alone do not establish event grounding, the authors say.
From the original article
1 min read at huggingface.co
Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions.
By Yinsong Yan, Ling Wang, Yuxuan Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.