PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving
PACE cuts perceived latency in retrieval-augmented dialogue serving via cascading routing and filler control, reaching 0.41s P95 under load.
PACE is a serving framework for retrieval-augmented dialogue that optimizes Perceived Time-to-First-Response (PTFR) under quality and cost constraints. It combines a load-adaptive cascading router, a joint path-filler controller, and volatility-aware cache admission, deployed on a humanoid-robot sales service. On 75k CarQA requests, the cascade halved pure-LLM P95 PTFR (0.29s vs 0.53s) and the adaptive controller reached 0.41s P95, 2.4x better than RAG at high load; filler calls dropped 94% and stale answers fell from 86% to 0%.
- Jointly controls answer source routing and filler content during waiting
- Halves P95 perceived latency versus pure-LLM serving on 75k CarQA requests
- Volatility-aware cache admission cuts stale answers from 86% to 0%
- First quantification of filler-answer conflict risk in deployed services
From the original article
1 min read at arxiv.orgWe present the PACE, a framework for retrieval-augmented dialogue serving that formalizes Perceived Time-to-First-Response (PTFR) as a QoE objective and minimizes it under quality/cost constraints. Unlike prior work on cascaded routing, semantic caching, or adaptive retrieval, PACE jointly controls which answer source composes the response and what fills the waiting window.
By Lin Huang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.