LLM Classification Is Feature Engineering
Argues LLM classifiers should feed a downstream logistic regression, yielding calibration, threshold control, and principled use of structured covariates.
The post contends that LLM-as-classifier setups suffer from poorly calibrated hard labels, opaque use of prompt context and structured data, and weak interpretability. Wrapping the LLM verdict as a feature in a logistic regression restores calibrated probabilities, precision–recall threshold control, and the ability to incorporate additional covariates. Further gains can come from more training data, richer features such as log probabilities and subverdicts, and swapping in downstream models like xgboost or neural networks. An irony detection test case illustrates the approach.
- Raw LLM verdicts are hard labels with poorly calibrated confidence, hindering precision–recall tuning
- Treating LLM output as one feature in logistic regression yields calibrated, threshold-controllable predictions
- Suggests more features (log probabilities, subverdicts) and flexible downstream models like xgboost
- Includes an irony detection test case demonstrating the framework
From the original article
14 min read at minimallysufficient.comLLMs-as-classifiers, prompts applied to a context and returning a label, suck to work with. This is especially painful because they often perform pretty decently. But let’s consider some of the things we’d want in a classifier and see how an LLM-as-classifier stacks up:
By minsufficient · minimallysufficient.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.