STRATA scores each step of temporal clinical hypothesis tests
Original title: Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses
The benchmark has 8,735 vaccine-report hypotheses; ART narrows a record-selection gap and brings a 31B model near frontier scores.
The paper says language models state temporal findings without a significance test, while existing data-analysis benchmarks grade only the conclusion. STRATA grades every decision toward a verdict on 8,735 hypotheses over clinical timelines of vaccine adverse-event reports, with gold plans, evidence sets, p-values, and verdicts from a released codebook and a deterministic oracle. ART (articulate, retrieve, test) exposes each decision. Experiments with five configurations find most failures in choosing which records count.
Unconfirmed: The 0.01 gap and where systems fail are the authors' experimental claims.
- Gold labels include plans, evidence sets, p-values, and verdicts.
- A deterministic oracle reproduces verdicts under the released codebook.
- Authors say frontier models succeed once given the codebook.
- Under ART, a 31B open model is within 0.01 of frontier models.
From the original article
1 min read at arxiv.orgAsk a language model whether sales tend to dip after a product recall, or whether one side effect tends to follow another after a vaccine, and it answers in a fluent sentence that reads like a finding, with no test behind it. Assertions about how events unfold across many cases fill the Web, and each is a temporal hypothesis that only a significance test over the right records can settle.
By Chengyang He · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.