Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models
An open methodology toolkit measures whether transformer language models contextualize fixed word forms across domains using bridge forms and layer-wise silhouette analysis.
The manual documents an open toolkit built around 'bridge forms' - identical written words recurring across two or more subject domains with a different sense in each - to test whether transformer language models individuate word occurrences by context beyond the embedding layer. It covers declarative specification of bridge forms, Wikipedia corpus acquisition, occurrence localization, layer-wise representation extraction, domain-pairwise silhouette measurement, and visualization, justifying each choice against failure modes such as sense contamination and subword-tokenization misalignment. It is a methodological and implementation reference and reports no empirical results.
- Introduces 'bridge form' construct to isolate context-driven representation change
- Pipeline spans Wikipedia corpora, layer-wise extraction, and silhouette separation metrics
- Documents pitfalls: sense contamination, multi-group silhouette bias, tokenization misalignment
- Methodology and implementation reference; no empirical model results reported
From the original article
1 min read at arxiv.orgA transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers. Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way.
By José Luciano Verçosa Marques · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.