TokenRouter: Efficient Serving System for Token-Level LLM Routing
TokenRouter raises token-level LLM routing decode throughput by 2.01–64.15× over existing serving systems.
Stories per day · 30 days
3 RSS
3 stories, newest first · one card per story
Stories tagged #batching
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Continuous depth batching enables efficient depth-adaptive inference for looped language models, reaching up to 99% of estimated maximum speedup.
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Study shows LLM test-time scaling cost depends on generation schedule: eight serial A100 calls use ~4.7x more GPU energy than one batched call.