Whitening Improves Robustness to Spurious Correlations in Linear Probes
Whitening linear-probe features reduces reliance on spurious correlations without labeled data or known confounders.
The paper studies linear probes fitted on pretrained representations and links them to max-margin classifiers that prefer large-eigenvalue directions of the covariance. Whitening equalizes those eigenvalues, reducing reliance on spurious correlations without labeled data or prior knowledge of the spurious features. Experiments on a synthetic data process and standard spurious-correlation benchmarks show improved robustness, including when whitening is added to existing methods.
- Linear probes favor high-eigenvalue covariance directions, which can be spurious.
- Whitening equalizes eigenvalues and reduces that bias without labels or known spurious features.
- Gains appear on a synthetic process and standard spurious-correlation benchmarks, including when combined with existing methods.
Full article138 words · extracted from arxiv.org · click to collapse
Deep neural networks tend to rely on simple features that may be spurious and thus fail to generalize. We study this problem in the setting of linear probes, where a (generalized) linear model is fitted on the representations of a (pretrained) model. We use the connection of these models to the max-margin classifier, and show they favor directions associated with large eigenvalues of the covariance matrix. Whitening removes this preference by equalizing the eigenvalues of the covariance matrix. This observation motivates whitening as a preprocessing step that can reduce reliance on spurious correlations without requiring prior knowledge of their presence or labeled data. We examine the effect of whitening on a synthetic data-generating process and standard spurious correlation benchmarks, and find that it improves robustness. We also find that whitening can improve robustness when added to existing approaches.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.39177