OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
Researchers introduce OmniVChat, a native audio-visual dialogue task with a synthesis studio, benchmark, and RL reward design that improves Qwen3-Omni-Instruct.
The OmniVChat paper defines native audio-visual dialogue where omni models simultaneously receive audio and video and reply without text prompts, external captioning, or speech recognition. The authors present OmniVChat-Studio, a multi-agent engine for synthesizing single- and multi-turn audio-visual dialogues, OmniVChat-Bench covering five ability categories, and OmniVChat-RL, a reinforcement learning reward jointly targeting correctness, efficiency, and style. Training Qwen3-Omni-Instruct with the method improves performance on both the synthetic benchmark and human-recorded OmniVChat-Bench-Human.
- OmniVChat-Studio synthesizes single- and multi-turn audio-visual training dialogues
- OmniVChat-Bench evaluates omni models across five ability categories
- OmniVChat-RL jointly optimizes reply correctness, efficiency, and style
- Qwen3-Omni-Instruct trained with the reward transfers to real human-recorded dialogues
- Tags
- audio-visual-dialoguebenchmarkdata-synthesismultimodalomni-modelsqwen3-omnireinforcement-learning
- Products
- OmniVChat-StudioOmniVChat-Bench
- AI models
- Qwen3-Omni-Instruct
From the original article
1 min read at huggingface.co
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition.
By Haolin He, Yunfei Chu, Qi Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.