Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency
Models pretrained on chronological knowledge update faster than shuffled generalists, and a derived law predicts that cost.
The paper studies how the organization of time-varying knowledge during pre-training affects later update efficiency. Models trained on chronologically ordered data, called specialists, focus on the most recent knowledge, while models trained on the same data in random order, called generalists, perform across all periods. Specialists require far fewer update steps to acquire new knowledge. The authors derive a law linking update cost to optimization dynamics, propose a data-organization strategy that preserves historical knowledge, and report that the benefit extends to real pretrained models and existing update tools.
- Chronological pre-training creates specialists; shuffled order creates generalists.
- Specialists need far fewer steps to learn new knowledge.
- A first-principles law predicts update cost from optimization dynamics.
- Effects carry over to real pretrained models and existing update tools.
From the original article
1 min read at arxiv.orgAn ideal (language) model must continually update its knowledge. Existing work largely focuses on making these updates efficient after training. We investigate how the organization of time-varying knowledge during pre-training shapes the efficiency of subsequent updates.
By Sheng Zhao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.