TokenRouter: Efficient Serving System for Token-Level LLM Routing
TokenRouter raises token-level LLM routing decode throughput by 2.01–64.15× over existing serving systems.
TokenRouter is a serving system for token-level LLM routing that separates request-centric routing logic from model-centric execution. Each model runs in a subserver with asynchronous dispatch and a delayed-batching scheduler tuned from a throughput model. Across routing algorithms, workloads, and model pairs, it reports 2.01–64.15× higher decoding throughput than existing systems, with code released publicly.
- Serving system for token-level routing across multiple LLMs.
- Request-centric programming with a subserver and async dispatch per model.
- Delayed-batching scheduler hyperparameters come from a throughput model.
- Decoding throughput is 2.01–64.15× higher than existing systems.
- Code is released at github.com/thu-nics/TokenRouter.
From the original article
1 min read at huggingface.co
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains.
By Tianyu Fu, Tengxuan Liu, Ruoxi Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.