Learning the Cost of Reliable Inference
A reverse-auction procurement platform learns LLM quality and price, finding 10–71% margins across Llama and Qwen tasks.
The paper proposes a procurement platform that prices LLM tokens through provider competition rather than a fixed per-token rate. A reverse second-price auction elicits truthful cost bids while the platform learns each provider's quality and routes queries to the cheapest provider above a quality threshold. Experiments on Llama and Qwen models using mathematical reasoning and question-answering benchmarks find pricing margins from 10 percent to 71 percent depending on task and quality threshold. The authors argue this exposes inefficiency in today's fixed-price inference market.
- Reverse second-price auction incentivizes providers to bid true average serving cost.
- Platform learns quality and routes to the cheapest provider above a threshold.
- Llama and Qwen experiments show margins from 10 percent to 71 percent.
- Results suggest fixed per-token pricing leaves substantial user savings uncaptured.
Full article207 words · extracted from arxiv.org · click to collapse
Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on these platforms typically use a fixed price per token, preventing users from achieving the most competitive price for their tasks. % workloads. In this work, we design a procurement platform where token prices for each task are driven by provider competition, enabling users to secure competitive pricing for guaranteed quality levels. To this end, the platform sequentially routes queries via a reverse second-price auction that incentivizes model providers to truthfully bid their best estimate of the average cost to serve a user's query. As it routes queries, the platform learns the quality offered by each provider and progressively routes queries to the most cost-competitive provider among those meeting a desired quality threshold. To validate our design, we conduct experiments with multiple LLMs from the \texttt{Llama} and \texttt{Qwen} families on popular mathematical reasoning and question-answering benchmarks. The results show that the pricing margin of the most cost-competitive provider on our platform varies significantly---from $10\%$ to $71\%$---depending on the task and quality threshold. This suggests a substantial inefficiency in the current fixed-price market, and it demonstrates that our platform may enable users to capture maximum savings whenever competitive market conditions permit.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.28322
Hacker News · AI