Omni Interaction Agent Technical Report
Researchers release Gander, an end-to-end omni interaction model with full-duplex streaming across video, speech, and text plus agentic capabilities.
Gander is an end-to-end model unifying omni perception, realtime interaction, and agentic capabilities in a single framework, accepting continuously streaming video, speech, and text. It uses a Cerebellum-Brain architecture where the Cerebellum handles realtime conversation and the Brain handles reasoning and agentic tasks, built on a streaming Thinker-Talker design with chunk-level token streams. Internal human evaluations report spoken dialogue on par with SOTA open source models and competitive omni interaction; the models, code, and data are released publicly.
- Cerebellum-Brain architecture splits realtime interaction from complex reasoning and agentic task execution.
- Streaming Thinker-Talker design flattens multimodal inputs and outputs into chunk-level token streams.
- Supports user interruption, proactive intermediate feedback, and follow-up questions in full duplex.
- Matches SOTA open source spoken dialogue quality; models, code, and data released.
From the original article
1 min read at huggingface.co
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework.
By Orantqing, Shengpeng Ji, Junlong Tong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.