How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
Researchers give a recipe for robust scaling-law comparisons of HEP models on ATLAS JetSet2.
The authors present a procedure for deriving scaling laws and comparing design choices for high-energy physics machine-learning tasks. After validating trajectories on toy problems, they apply it to multi-task transformers on the roughly 11-billion-jet ATLAS JetSet2 dataset in compute- and data-constrained regimes. At compute-optimal scaling they recover a near-equal square-root-of-compute dependence of model and dataset size, and they predict jointly optimal model size, training horizon, learning rate, and batch size under early stopping. Auxiliary objectives lower the primary jet-classification loss at equal compute.
- Procedure derives robust scaling laws for HEP machine-learning tasks.
- Applied to multi-task transformers on ~11-billion-jet ATLAS JetSet2.
- Predicts joint optima for size, horizon, learning rate, and batch size.
- Compute-optimal scaling shows near-equal square-root dependence of model and data.
From the original article
1 min read at arxiv.orgMuch of the recent progress in machine learning domains such as language models has come from scaling laws that predict performance as a function of training effort. In high-energy physics (HEP) similar behavior has now been observed. To aid further study, we present a systematic procedure to derive robust scaling laws and compare design choices on the relevant budget axes for HEP tasks.
By Matthias Vigl · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.