Point-in-time models tie temporal variability to later pretraining gains
Original title: What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
High-TVS items beat low-TVS items in 34 of 36 settings by 7.3 chance-normalized accuracy points on average.
The authors define a temporal variability score as the coefficient of variation of an item's negative log-likelihood across point-in-time models trained on yearly web snapshots. They test six benchmarks with six models from multiple open-source families. Low- and high-TVS items are learned similarly early, but high-TVS items improve faster later. The end-of-run advantage is strongest on knowledge-intensive tasks, including comparisons of items with similar difficulty.
Unconfirmed: The paper does not name the six models or benchmarks in this abstract.
- TVS is the coefficient of variation of negative log-likelihood.
- A higher score means less consistent representation across years of web text.
- Early training treats low- and high-TVS items similarly.
- The pattern remains when items of similar difficulty are compared.
From the original article
1 min read at arxiv.orgLanguage models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time.
By Chang Su · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.