Heartbeat Thinking keeps voice-agent tools running while listening
Original title: Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents
On FDBench v3, tool coverage rises 16 percentage points and mean final-response latency falls 36.4 percent versus a windowed baseline.
Heartbeat Thinking lets realtime voice agents keep reasoning and tool calls running while the user is still speaking. New Thinker invocations read the latest partial transcript and a shared record of completed work, instead of truncating an in-flight tool call when the transcript updates. Evaluations on SpeechFCEval, FDBench v3, and AudioCRAG report better speculative tool coverage or retrieval reuse than a SHANKS-style windowed baseline, with effects on final task quality that depend on the benchmark.
Unconfirmed: Final task-quality effects are benchmark-dependent and not uniformly improved.
- Ongoing invocations and dispatched tools are allowed to finish.
- Benchmarks are SpeechFCEval, FDBench v3, and AudioCRAG.
- FDBench v3 tool coverage increases by 16 percentage points.
- Mean final-response latency on FDBench v3 falls by 36.4 percent.
From the original article
1 min read at huggingface.co
Tool-using voice agents often leave users waiting after speech ends, even when actionable information arrives earlier in the request. Speculative approaches begin work during speech, but chunk-bounded or interrupt-on-update generation can truncate a tool call before its arguments are complete. This couples computation time to speech updates, limiting useful work completed in advance.
By Zhenyu Zhang, Kewei Fu, Yongqi Tong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.