Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution
Speculative tool execution cuts on-device voice-agent median time-to-first-audio from 5.79s to 4.60s.
Tool-augmented speech assistants usually serialize speech recognition, language-model inference, and tool execution, so tool latency begins only after the user stops speaking. The authors predict tool requests from partial ASR hypotheses, run them speculatively, and inject cached results into the LLM prompt. A rule-based check drops caches invalidated by self-corrections, and the model can still call tools directly so worst-case latency stays within the serial baseline. Live Android measurements reduce median time-to-first-audio from 5.79 seconds to 4.60 seconds and standard deviation from 3.49 to 2.81 seconds.
- Predictor starts tool calls from partial ASR hypotheses during speech.
- Rule-based validation filters caches invalidated by user self-corrections.
- LLM fallback keeps worst-case latency within the serial pipeline.
- Android tests cut median time-to-first-audio by about 1.2 seconds.
From the original article
1 min read at huggingface.co
Tool-augmented speech assistants typically serialize automatic speech recognition, large language model inference, and external tool execution. As a result, tool latency is incurred only after the user has finished speaking and the LLM has identified the required tool calls.
By Kyudan Jung, Hyunsin Park, Yoonhyung Lee · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.