Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
French BabyLM entry METRON-FR (125M GPT-2, 92.47M words) shows tokenizer artifacts dominate child-scale zero-shot evaluation; proposes standard diagnostics.
METRON-FR is a 125M-parameter GPT-2 pretrained on 92.47M French words, submitted to the BabyLM 2026 Strict track, scoring 85.97% on the native Quebec-French QFrBLiMP benchmark and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE protocol combining French task-data translation with rank-16 LoRA shows relational tasks gain while world-knowledge tasks regress. Bilingual Lexicon Induction reaches p@1 of 68.84%, 18x above chance, and ablations show single-token zero-shot scoring is dominated by tokenizer and template artifacts at child scale.
- 125M GPT-2 trained on 92.47M French words for BabyLM 2026 Strict track.
- Scores 85.97% on QFrBLiMP and 62.80% on BabyLM weighted leaderboard.
- Cross-lingual GLUE with rank-16 LoRA: relational tasks gain, world-knowledge tasks regress.
- Tokenizer and template artifacts dominate single-token zero-shot scoring at child scale.
- Recommends tokenizer-swap sensitivity and native-language minimal-pair benchmarks as diagnostics.
From the original article
1 min read at arxiv.orgWe submit MéTRON-FR, a 125M GPT-2 pretrained on 92. 47M words of French, to the BabyLM 2026 Strict track. It scores 85.
By Adam Zachary Wasserman · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.