Zero-shot Jev replaces predictors on 12 of 46 NLP test sets
Original title: Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers
With 500 labels, prompt optimization closed a median 72% of the gap; LoRA on Qwen3.8-27B closed 97%.
Authors swap the predictors of 24 ACL-venue papers for TypeSafe’s zero-shot Typed Decision Model Jev, keeping each paper’s splits, labels, and metric, and check results with Cloudflare’s Clef, Microsoft-Decision-1, and OpenAI’s GPT-6 Luna Decisions. The 46 English test sets form the TDM-Gauntlet benchmark. A zero-shot prompt replaced the paper’s predictor on 12 of 46 sets, mostly plain single judgments. Where it lagged, shortfalls were often tied to unstated annotation conventions; calibration and GEPA-style prompt optimization with 500 labels closed a median 72% of the gap.
- Off-the-shelf GEPA closed about half as much of the gap as the authors’ search.
- Median search cost was $1.66, with no user-managed GPU.
- Fine-tuned RoBERTa and DeBERTa closed 59% of the gap on the same labels.
- LoRA fine-tuning of Qwen3.8-27B closed 97%, remaining the most accurate option.
- Products
- ClefMicrosoft-Decision-1
From the original article
1 min read at arxiv.orgTyped Decision Models such as TypeSafe's Jev, also called System One or System 1 models, answer typed questions about an input with a probability for each option, and they need no training. Many NLP papers still fine-tune a task-specific classifier, and prompted large language models (LLMs) trail such classifiers on most text classification benchmarks.
By Marek Šuppa · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.