Skip to content
NewA calmer, faster ZeroHour

Zero-shot Jev replaces predictors on 12 of 46 NLP test sets

Original title: Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers

arXiv cs.AI / cs.LG / cs.CLby Marek Šuppa ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

With 500 labels, prompt optimization closed a median 72% of the gap; LoRA on Qwen3.8-27B closed 97%.

Authors swap the predictors of 24 ACL-venue papers for TypeSafe’s zero-shot Typed Decision Model Jev, keeping each paper’s splits, labels, and metric, and check results with Cloudflare’s Clef, Microsoft-Decision-1, and OpenAI’s GPT-6 Luna Decisions. The 46 English test sets form the TDM-Gauntlet benchmark. A zero-shot prompt replaced the paper’s predictor on 12 of 46 sets, mostly plain single judgments. Where it lagged, shortfalls were often tied to unstated annotation conventions; calibration and GEPA-style prompt optimization with 500 labels closed a median 72% of the gap.

Key facts
  • Off-the-shelf GEPA closed about half as much of the gap as the authors’ search.
  • Median search cost was $1.66, with no user-managed GPU.
  • Fine-tuned RoBERTa and DeBERTa closed 59% of the gap on the same labels.
  • LoRA fine-tuning of Qwen3.8-27B closed 97%, remaining the most accurate option.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Typed Decision Models such as TypeSafe's Jev, also called System One or System 1 models, answer typed questions about an input with a probability for each option, and they need no training. Many NLP papers still fine-tune a task-specific classifier, and prompted large language models (LLMs) trail such classifiers on most text classification benchmarks.

arXiv cs.AI / cs.LG / cs.CL

By Marek Šuppa · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.