Ask a language model whether a particular side effect tends to follow another after a vaccination, and it will often respond with a confident, fluently worded sentence that reads like an established finding — even though no actual statistical test backs it up. Claims about how events unfold over time are common online, yet each one constitutes a temporal hypothesis that can only be settled through a proper significance test applied to the right data. Existing benchmarks for evaluating AI-driven data analysis typically grade only the final conclusion, handing the model the relevant records in advance, which conceals errors in test selection, null-hypothesis framing, or evidence selection.
A research team introduces STRATA, a benchmark designed to assess every decision leading up to a model's verdict, not just the verdict itself. It draws on 8,735 hypotheses built from clinical timelines in vaccine adverse-event reports. Each hypothesis comes with a gold-standard analysis plan, a validated evidence set, p-values, and a verdict, all derived from a publicly released codebook and reproduced through a deterministic oracle to ensure reliable ground truth.
The authors also propose ART (articulate, retrieve, test), a staged paradigm that makes each step of a model's reasoning separately inspectable: formulating the hypothesis, retrieving the relevant evidence, and running the statistical test. This decomposition makes it possible to pinpoint exactly where a system goes wrong, rather than simply checking whether the final answer happens to be correct.
Experiments across five model configurations reveal that most failures stem from selecting which records actually bear on the hypothesis — a gap between the verdict produced and the evidence genuinely used, a gap that plain accuracy metrics fail to expose. The ART paradigm closes most of this gap and brings a 31-billion-parameter open model within 0.01 of frontier model performance under the same evaluation setup. Frontier models themselves only reach this level of reliability when explicitly given the reference codebook.
The findings highlight a point often overlooked in evaluating AI systems for healthcare applications: a correct answer can mask flawed reasoning underneath, a real concern in a domain where a wrong conclusion carries tangible costs.