MMDM benchmark tests 28 models on 5,448 decision tasks
Original title: MMDM: Measuring Massive Multitask Decision Making
Best hard multi-select accuracy is 36.81%; Decider-V1.2 falls from 78.75% on world knowledge to 47.95% on constraint planning.
MMDM (Massive Multitask Decision Making) is a benchmark for System-1 decision models that pick structured options in one forward pass without generating reasoning text. It has 5,448 questions in 16 categories, mixing new tasks with rewritten source-grounded items for single-select, multi-select, and soft-label decisions. The authors evaluate 28 models and report strong common-knowledge scores but weaker results on interdependent options, constraints, and multi-select recovery.
- Option spaces range from 2 to 1,024; inputs reach 64,817 tokens.
- Perplexity Decider-V1.2 scores 53.95% on option logic.
- Tasks include agent state tracking, tool selection, preferences, and rubric scoring.
- Organizations
- Perplexity
- AI models
- Perplexity Decider-V1.2
From the original article
1 min read at huggingface.co
System-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization.
By Junhao Shen, Qing Zong, Yue Guo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.