Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Synthetic pre-pretraining still saves tokens up to 7B parameters, but gains come from retrieval, not grammar.
1 stories, newest first · one card per story
Synthetic pre-pretraining still saves tokens up to 7B parameters, but gains come from retrieval, not grammar.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views