OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
OmniSeek trains Omni-LLMs to seek audio-visual evidence across turns using a 170K trajectory corpus.
OmniSeek turns an Omni large language model into a multi-turn agent that decides whether to look or listen and over which temporal window. Retrieved audio or visual segments are appended back into context for later reasoning. A data engine builds OmniTraj-170K, a corpus of multi-hop chain-of-thought trajectories with interleaved evidence, used for supervised cold-start and two-stage reinforcement learning. An Audio-Visual Necessity objective rewards trajectories that depend on both modalities.
- OmniSeek lets an Omni-LLM choose when and where to look or listen.
- OmniTraj-170K supplies multi-hop trajectories with interleaved audio and visual evidence.
- Training uses supervised tool use, then two-stage RL with verifiable rewards.
- An Audio-Visual Necessity objective penalizes single-modality shortcuts.
From the original article
1 min read at huggingface.co
We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use.
By Haibo Wang, Jiteng Mu, Jialu Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.