Type Diversity Enables Transformers to Generalise Compositionally
Researchers show lexical-versus-structural compositional generalization gaps in Transformers stem from type diversity imbalance in datasets, not architectural limits.
The paper argues that Transformers' difficulty with structural compositional generalization is an artifact of low structural type diversity in prior benchmark datasets rather than an architectural limitation. Using Grammatical Framework, the authors create linguistically diverse variants of COGS and SLOG. They find type diversity correlates with compositional generalization equally in lexical and structural test cases, contradicting previous claims that compound divergence explains task difficulty.
- Type diversity, not architecture, explains lexical vs structural generalization gap
- Linguistically diverse COGS and SLOG variants built with Grammatical Framework
- Type diversity correlates equally with lexical and structural generalization
- Contradicts prior compound-divergence explanation for compositional task difficulty
From the original article
1 min read at arxiv.orgCompositional generalisation has been divided into lexical and structural generalisation. Previous work has found that structural generalisation is harder than lexical for Transformers. We propose that this difference is not inherent to Transformers, but due to the high diversity of lexical types and low diversity of structural types in the specific datasets of these previous works.
By Anssi Moisio · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.