Skip to content
NewA calmer, faster ZeroHour
StoryAI industryinfo

Study swaps decision models into 24 ACL papers' predictors

2 sources · 2 articlesFirst reported Updated ()1
What's new

First merged summary.

Merged summary · x-ai/grok-4.7 · rewritten as coverage arrivesSummaries by Vani AI

A zero-shot decision model matched paper predictors on 12 of 46 sets; LoRA still closed more of the gap.

Authors replaced the predictors in 24 ACL-venue papers with TypeSafe’s zero-shot Typed Decision Model Jev, keeping each paper’s splits, labels, and metric. Results were checked with Cloudflare’s Clef, Microsoft-Decision-1, and OpenAI’s GPT-6 Luna Decisions. The 46 English test sets form the TDM-Gauntlet benchmark. A zero-shot prompt replaced the paper’s predictor on 12 of 46 sets, mostly plain single judgments. Where it lagged, shortfalls were often tied to unstated annotation conventions; calibration and GEPA-style prompt optimization with 500 labels closed a median 72% of the gap. LoRA fine-tuning of Qwen3.8-27B closed 97% and remained the most accurate option.

Key facts
  • Zero-shot prompt replaced the paper predictor on 12 of 46 English sets.
  • Prompt optimization with 500 labels closed a median 72% of the gap.
  • Off-the-shelf GEPA closed about half as much as the authors’ search.
  • Median search cost was $1.66, with no user-managed GPU.
  • Fine-tuned RoBERTa and DeBERTa closed 59% of the gap on the same labels.
  • LoRA on Qwen3.8-27B closed 97% of the gap.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

Timeline

dated events in the reporting · oldest first
  1. Nov 15, 2026Paper reports TDM-Gauntlet results on 46 English sets. · arXiv

What each outlet adds

beyond the shared facts
  • arXivBenchmark gaps, search cost, and LoRA versus prompt optimization.

Coverage

oldest first · one row per article · UTC
  1. · 2d ago arXiv cs.AI / cs.LG / cs.CLFirst report

    Zero-shot Jev replaces predictors on 12 of 46 NLP test sets

    With 500 labels, prompt optimization closed a median 72% of the gap; LoRA on Qwen3.8-27B closed 97%.

  2. · 1d ago TechCrunch · AI

    Microsoft rolls out Decision 1.1 as decision-model use grows

    Vercel says more than one in four October router requests went to decision models; Sqwish raised $3 million.