Omni-Streaming Thinking
Omni-Streaming Thinking fixes premature cross-modal commitment in streaming omni-modal models via pending claims verified against modality-specific evidence, beating baselines by over 10%.
The paper identifies 'premature cross-modal commitment', where streaming models keep relaying early visual interpretations even after audio contradicts them. OST generates evidence-linked pending claims with future verification intervals, stores audio and visual evidence separately, and refutes claims when contradictory evidence appears. Built on a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, it outperforms open baselines by more than 10% relative on five streaming and audio-visual benchmarks. On the new OST-DiagBench it reaches d-prime 2.95 versus at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.
- Addresses premature cross-modal commitment in streaming omni-modal models
- Pending claims verified against modality-specific evidence at set intervals
- Outperforms open baselines by over 10% on five benchmarks
- D-prime 2.95 on new OST-DiagBench vs 1.38 for best baseline
From the original article
1 min read at huggingface.co
Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it.
By Enjun Du, Siyi Liu, Ziyu Zheng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.