NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
NemotronLabs VoiceChat is an open full-duplex speech-to-speech model that calls tools during real-time conversation.
NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel text and function-call outputs, an RNN-T transcription branch, and a streaming TTS decoder. On Full-Duplex-Bench 1.0 it reports the lowest pause-handling takeover among open-weight systems, 100% takeover after interruptions, and 4.33/5 response quality. It resumes after backchannels in 93% of Full-Duplex-Bench 1.5 cases, scores 55.1 on VoiceBench, and reaches 82.5% tool-selection F1 on Full-Duplex-Bench 3.0, while argument accuracy and end-to-end execution still lag.
- Open streaming speech-to-speech model with native tool calling.
- Lowest pause-handling takeover on Full-Duplex-Bench 1.0 among open-weight systems.
- 100% interruption takeover and 4.33/5 post-interruption response quality.
- VoiceBench score 55.1; 82.5% tool-selection F1 on Full-Duplex-Bench 3.0.
- Argument accuracy and end-to-end tool execution remain weaker.
- Organizations
- NemotronLabs
- AI models
- NemotronLabs VoiceChat
From the original article
1 min read at arxiv.orgWe introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder.
By Jagadeesh Balam · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.