A Better Metric Will Not Save You: What You Compare Beats How You Compare It
Representational similarity rankings depend more on which conditions are compared than on the similarity metric.
The paper argues that RSA, CKA, and Procrustes treat some transformations as irrelevant but cannot correct earlier choices about which inputs are admitted and how they are grouped. Using vowel acoustics across fifteen languages and twenty-seven representations, where the target organization is known independently, changing which categories are compared at a fixed count shifts the gap between two representations by more than twice the measured effect. Changing the number of categories does not. Model rankings are stable under these choices within a setting, but rankings across languages are not.
- Condition selection precedes any invariance of RSA, CKA, or Procrustes.
- Category choice moves representation gaps by more than twice the measured effect.
- Changing how many categories are compared does not move that gap.
- Model rankings hold within a setting but not across languages.
From the original article
1 min read at arxiv.orgRepresentational similarity measures (RSA, CKA, Procrustes) are chosen for the transformations they treat as irrelevant, and all of them take the set of conditions as given. That set is fixed by two decisions, which inputs are admitted and how they are grouped, both taken before the matrix a measure operates on exists: no invariance property reaches them.
By Noureddine Khaous · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.