Learning the Cost of Reliable Inference
A reverse-auction procurement platform learns LLM quality and price, finding 10–71% margins across Llama and Qwen tasks.
The paper proposes a procurement platform that prices LLM tokens through provider competition rather than a fixed per-token rate. A reverse second-price auction elicits truthful cost bids while the platform learns each provider's quality and routes queries to the cheapest provider above a quality threshold. Experiments on Llama and Qwen models using mathematical reasoning and question-answering benchmarks find pricing margins from 10 percent to 71 percent depending on task and quality threshold. The authors argue this exposes inefficiency in today's fixed-price inference market.
- Reverse second-price auction incentivizes providers to bid true average serving cost.
- Platform learns quality and routes to the cheapest provider above a threshold.
- Llama and Qwen experiments show margins from 10 percent to 71 percent.
- Results suggest fixed per-token pricing leaves substantial user savings uncaptured.
From the original article
1 min read at arxiv.orgBenchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on these platforms typically use a fixed price per token, preventing users from achieving the most competitive price for their tasks. % workloads.
By Dimitrios Rontogiannis · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.