Type Diversity Enables Transformers to Generalise Compositionally
Researchers show lexical-versus-structural compositional generalization gaps in Transformers stem from type diversity imbalance in datasets, not architectural limits.
The paper argues that Transformers' difficulty with structural compositional generalization is an artifact of low structural type diversity in prior benchmark datasets rather than an architectural limitation. Using Grammatical Framework, the authors create linguistically diverse variants of COGS and SLOG. They find type diversity correlates with compositional generalization equally in lexical and structural test cases, contradicting previous claims that compound divergence explains task difficulty.
20