Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models
Audits of 10 classifiers on BRFSS show target leakage, not model class, drives the reported 0.89 AUROC in survey-based cardiovascular screening.
The study benchmarks ten model classes, including glass-box and tabular foundation models, for prevalent myocardial infarction on 442,067 respondents of the 2022 BRFSS across five feature tiers of decreasing leakage risk. Removing two post-diagnostic features costs every model 0.049-0.051 AUROC and collapses performance into a 0.0045-wide band, and the explainable boosting machine matches all alternatives within 0.005 while scoring roughly 104x faster than the strongest foundation model. Frozen models transport within 0.002 AUROC to 2023 data; the authors conclude evaluation practice and feature sets, not model capacity, are the binding constraint.
- Reported ~0.89 AUROC in survey-based screening is attributed to target leakage.
- 442,067 respondents (2022 BRFSS) audited for discrimination, calibration, fairness, and coverage.
- Explainable boosting machine non-inferior to tabular foundation models and ~104x faster.
- Gender detection gap (75.4% vs 89.0%) reduced to 0.010 via shape-function edits.
- Mondrian conformal calibration repaired coverage disparities across strata.
From the original article
1 min read at arxiv.orgCardiovascular screening models trained on national health surveys routinely report areas under the receiver operating characteristic curve (AUROC) near 0. 89. We asked whether that accuracy reflects learning or target leakage, whether tabular foundation models change the answer, and whether the properties deployment requires survive joint examination.
By Raad Bin Tareaf · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.