WORTH pilot tests models on mystery evidence discovery
Original title: WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot
In 37 scorable Episode 02 runs, 31 chose the same wrong participants; adjacent clue types were selected 25 times and 0 times.
WORTH asks models to read unannotated mystery narratives, select key clue sentences, identify participants, and explain their reasoning. A two-episode full-story pilot ran 70 formal model runs, 67 of them content-scorable. Clue hits and participant judgments did not move together. The authors say the pilot does not establish stable rankings or general mechanisms, and this version does not report human-model comparisons because human records are not yet cleaned.
Unconfirmed: Authors explicitly say the pilot does not support stable model rankings or general mechanism claims.
- Professional authors designed narratives that delay revelation of the truth.
- Natural-language explanations are used for diagnosis and are not scored alone.
- Episode 02 had 37 content-scorable runs.
- Human player records have not yet been cleaned.
From the original article
1 min read at huggingface.co
Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously and how information presentation within narratives guides or obscures judgment.
By Bo Zou, Chao Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.