ScAn-Bench: Evaluating Scaling Analysis Methodology
ScAn-Bench evaluates how researchers fit and extrapolate scaling laws using thousands of LLM and VLM checkpoints.
ScAn-Bench is a pair of surrogate benchmarks, ScAn-Bench-LLM and ScAn-Bench-VLM, built from 4,524 language-model and 8,024 vision-language-model pipeline checkpoints. The work systematically evaluates scaling-analysis methodology, covering both how scaling data are acquired and how laws and prescriptions are extrapolated across modalities. It targets a gap: scaling laws guide architecture, data, and hyperparameter choices, but methods for deriving them have lacked comparative evaluation.
- Surrogate benchmarks cover LLM and vision-language pipelines.
- Built from 4,524 LLM and 8,024 VLM checkpoints.
- Evaluates data acquisition and extrapolation methods for scaling laws.
- Compares methodology across language and vision-language modalities.
From the original article
1 min read at arxiv.orgRecent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types.
By Artin Sermaxhaj · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.