Embedding Models Measure in Peculiar Ways
Researchers find embedding models only weakly represent physical measurements like mass, distance, time, and volume, dominated by superficial string similarity.
The study tests whether embedding spaces capture objective notions of semantic equivalence and distance defined by physical measurements of mass, distance, time, and volume. Physical measurement is found to be only weakly modeled in embedding space, with representations strongly influenced by superficial string similarity, and recalibrating similarity metrics does not substantially improve alignment.
- Embedding distance poorly tracks objective physical measurement equivalence
- Superficial string similarity strongly shapes how measurements are represented
- Recalibration of similarity metrics yields little improvement
Full article83 words · extracted from arxiv.org · click to collapse
Embedding spaces define notions of semantic similarity and distance. We study whether those embeddings reflect physical measurements of mass, distance, time and volume, which admit a unique, objective notion of semantic equivalence and distance. We find that physical measurement is only weakly modeled in the embedding space, and that instead quite peculiar measurement patterns can be observed. Further analysis indicates that embedding representations of physical measurements are strongly influenced by superficial string similarity, and recalibration of similarity does not substantially improve the alignment.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.20821