Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Study shows visually grounded token embeddings in a small masked LM persist through training and improve object-property knowledge, but escape standard BabyLM benchmarks.
The paper implements ostensive definition for a small DeBERTa masked language model trained on 10M words, seeding visually grounded tokens with embeddings derived from labeled image regions before training. Visual initialization leaves a persistent, seed-replicated advantage on object-property knowledge (COMPS) and a corpus-tailored Visual-Property Swap benchmark covering color, material, size, and shape, but has no effect on most BabyLM grammar benchmarks. Synthetic grounding of previously unseeded words causally transfers the advantage to exactly those words.
- DeBERTa trained on 10M words with visually seeded embeddings
- Gains appear on COMPS and Visual-Property Swap object-property tests
- Standard BabyLM grammar benchmarks show no visual-initialization effect
- Synthetic grounding causally transfers advantage to newly seeded words
- Products
- COMPSVisual-Property Swap
- AI models
- DeBERTa
From the original article
1 min read at arxiv.orgA language model normally begins training with random word embeddings: whatever 'banana' means must be learned from training corpora. I implement St. Augustine's picture of word learning, meaning by ostension, for a small masked language model (DeBERTa) trained on 10M words: before training, visually grounded tokens receive embeddings derived from the image regions they label; other tokens start random.
By Lisa Bylinina · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.