OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue
Researchers introduce OmniVChat, a native audio-visual dialogue task with a synthesis studio, benchmark, and RL reward design that improves Qwen3-Omni-Instruct.
The OmniVChat paper defines native audio-visual dialogue where omni models simultaneously receive audio and video and reply without text prompts, external captioning, or speech recognition. The authors present OmniVChat-Studio, a multi-agent engine for synthesizing single- and multi-turn audio-visual dialogues, OmniVChat-Bench covering five ability categories, and OmniVChat-RL, a reinforcement learning reward jointly targeting correctness, efficiency, and style. Training Qwen3-Omni-Instruct with the method improves performance on both the synthetic benchmark and human-recorded OmniVChat-Bench-Human.
- OmniVChat-Studio synthesizes single- and multi-turn audio-visual training dialogues
- OmniVChat-Bench evaluates omni models across five ability categories
- OmniVChat-RL jointly optimizes reply correctness, efficiency, and style
- Qwen3-Omni-Instruct trained with the reward transfers to real human-recorded dialogues
Full article234 words · extracted from huggingface.co · click to collapse
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.21465