LLM classifier circuits raise accuracy at a fixed query cost
Original title: Circuits of LLM Classifiers
On seven tasks spanning safety, health, and reasoning, the method beats routing and cascading baselines.
The paper treats prompted language-model classifiers as noisy measurements of a target label and designs circuits that adaptively order queries and decisions. The goal is higher average accuracy at a given average query cost, with a characterization of the optimal circuit and accuracy limits. Across seven tasks covering safety, health, political science, preference, agent evaluation, decision-making, and reasoning, the method improves the cost-accuracy frontier versus routing and cascading.
- Predictions from prompted models are treated as noisy and unevenly reliable.
- The optimal circuit and fundamental accuracy limits are characterized.
- Practical guidance is given for realizing that circuit.
- Tasks also include human preference and agent evaluation.
From the original article
1 min read at arxiv.orgLanguage models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability?
By Hadi Khalaf · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.