Cost-aware abstention cuts wrong routes on IT ticket classification
Original title: Cost-Aware Abstention for LLM-Based IT Ticket Classification
On 47,837 tickets, a calibration-selected TF-IDF policy cuts test cost 38.1% and wrong routes from 15.3% to 3.5%.
The paper evaluates offline, cost-conditional abstention for routing an eight-class public IT-ticket corpus. Policies are chosen on calibration data under explicit costs and then frozen for one held-out test. Under costs (0, 1, 5), TF-IDF logistic regression with threshold 0.60 lowers test cost from 0.763 to 0.472. The best prompted setup reaches 0.477 accuracy on a stratified 1,000-ticket subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. The authors present an evaluation framework, not a new abstention algorithm.
Unconfirmed: Results are scoped to this corpus, prompts, model endpoints, and illustrative costs.
- The 95% CI for the 38.1% relative cost reduction is [35.7%, 40.4%].
- Frozen policies beat always-route and always-defer in 14 of 16 post-hoc cost settings.
- At break-even probability 0.80, seven prompted setups defer every ticket.
- Haiku few-shot routes only 19 of 1,000 tickets at that threshold.
- GPT-4o-mini few-shot has correctness AUROC 0.500 before and after isotonic recalibration.
From the original article
1 min read at arxiv.orgWrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data.
By Sachin Gupta · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.