Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Synthetic pre-pretraining still saves tokens up to 7B parameters, but gains come from retrieval, not grammar.
The authors test synthetic pre-pretraining (PPT) far beyond prior 1B-parameter and sub-2B-token studies, covering five PPT tasks, four pretraining mixtures, scales from 500M to 7B, and budgets up to 100B tokens. Token-efficiency and downstream gains persist at scale, including at least 21B tokens saved at 3B. They find no consistent link to a grammatical prior; gains instead come from PPT tasks that improve long-range retrieval. Gains hold across realistic mixtures of web, code, and math text, and weaken only when web text is removed.
- Study spans five PPT tasks, four data mixtures, 500M–7B models, and up to 100B tokens.
- At 3B scale, PPT saves at least 21B pretraining tokens.
- Gains track long-range retrieval, not grammatical acceptability.
- Benefits persist across mixtures and fade only when web text is absent.
From the original article
1 min read at huggingface.co
Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical prior, i. e.
By Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.