Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Study shows visually grounded token embeddings in a small masked LM persist through training and improve object-property knowledge, but escape standard BabyLM benchmarks.
The paper implements ostensive definition for a small DeBERTa masked language model trained on 10M words, seeding visually grounded tokens with embeddings derived from labeled image regions before training. Visual initialization leaves a persistent, seed-replicated advantage on object-property knowledge (COMPS) and a corpus-tailored Visual-Property Swap benchmark covering color, material, size, and shape, but has no effect on most BabyLM grammar benchmarks. Synthetic grounding of previously unseeded words causally transfers the advantage to exactly those words.
20