Unigram cross-entropy tracks zero-shot scores from pre-training overlap
Original title: The Lexical Imprint of Pre-training Data
Across 10 benchmarks, four corpora, and three scales, lower UCE predicted higher accuracy, and document selection and prompt routing also gained.
The paper defines word-level unigram cross-entropy as a tokenizer-agnostic, model-free measure of lexical alignment between a pre-training corpus and a benchmark. In from-scratch runs that change only the corpus, lower UCE consistently tracked higher zero-shot accuracy on every benchmark tested. Selecting documents to maximize alignment produced the predicted accuracy gains. Routing each prompt to the drafter whose domain corpus was most lexically aligned also beat a generalist-drafter baseline on throughput.
Unconfirmed: Findings come from the authors' controlled pre-training sweep and are not independently confirmed here.
- The controlled sweep used 10 zero-shot benchmarks, four corpora, and three model scales.
- Corpora more aligned with a benchmark, meaning lower UCE, scored higher.
- Document-level selection strengthened the lexical imprint and raised accuracy.
- Per-prompt routing to the most aligned domain drafter improved throughput.
From the original article
1 min read at huggingface.co
How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a target benchmark.
By Jeonghoon Kim, Woojin Chung, Woomin Song · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.