Paper gives an exact finite-sample identity for CV variance
Original title: A Theory of Cross-Validation Variance
K-fold is first-order variance optimal, leave-one-out is smallest among K-fold schemes, and the estimate is asymptotically normal.
The paper derives an exact finite-sample identity for the variance of a cross-validated performance estimate from how random splits weight each observation as training data and as a held-out evaluation. For a fixed training-set size, variance depends on those weights plus constants of the model, metric, and population, covering observation-averaged metrics such as mean squared error and pair metrics such as AUC. The identity splits sampling variance from extra variance caused by random fold assignment. The authors conclude K-fold is variance optimal at first order, its first-order variance does not depend on K, leave-one-out has the smallest variance among K-fold schemes under stated conditions, and leave-pair-out plays that role for AUC. They also show asymptotic normality with the identity's full variance.
- Monte Carlo and bootstrap cross-validation are not first-order variance optimal.
- K-fold's first-order variance does not depend on K.
- Leave-pair-out is the AUC analogue of leave-one-out.
- The identity covers both MSE-style metrics and AUC.
From the original article
1 min read at arxiv.orgCross-validation estimates predictive performance by repeatedly splitting the data into training and validation sets, fitting the model on the first and evaluating it on the second. Every observation influences the estimate in two roles, through its own evaluation when it is held out and through the models it helps fit when it trains. The random splits determine its weight in each role.
By Nathaniel S. O'Connell · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.