Study swaps decision models into 24 ACL papers' predictors
First merged summary.
A zero-shot decision model matched paper predictors on 12 of 46 sets; LoRA still closed more of the gap.
Authors replaced the predictors in 24 ACL-venue papers with TypeSafe’s zero-shot Typed Decision Model Jev, keeping each paper’s splits, labels, and metric. Results were checked with Cloudflare’s Clef, Microsoft-Decision-1, and OpenAI’s GPT-6 Luna Decisions. The 46 English test sets form the TDM-Gauntlet benchmark. A zero-shot prompt replaced the paper’s predictor on 12 of 46 sets, mostly plain single judgments. Where it lagged, shortfalls were often tied to unstated annotation conventions; calibration and GEPA-style prompt optimization with 500 labels closed a median 72% of the gap. LoRA fine-tuning of Qwen3.8-27B closed 97% and remained the most accurate option.
- Zero-shot prompt replaced the paper predictor on 12 of 46 English sets.
- Prompt optimization with 500 labels closed a median 72% of the gap.
- Off-the-shelf GEPA closed about half as much as the authors’ search.
- Median search cost was $1.66, with no user-managed GPU.
- Fine-tuned RoBERTa and DeBERTa closed 59% of the gap on the same labels.
- LoRA on Qwen3.8-27B closed 97% of the gap.
Timeline
dated events in the reporting · oldest first- Nov 15, 2026Paper reports TDM-Gauntlet results on 46 English sets. · arXiv
What each outlet adds
beyond the shared facts- arXivBenchmark gaps, search cost, and LoRA versus prompt optimization.
- Products
- ClefMicrosoft-Decision-1
- Organizations
- VercelSqwishOpenRouter
Coverage
oldest first · one row per article · UTCZero-shot Jev replaces predictors on 12 of 46 NLP test sets
With 500 labels, prompt optimization closed a median 72% of the gap; LoRA on Qwen3.8-27B closed 97%.
Microsoft rolls out Decision 1.1 as decision-model use grows
Vercel says more than one in four October router requests went to decision models; Sqwish raised $3 million.