How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Study of 800 models finds wild AI web text initially helps data-starved pretraining, then raises human-text loss.
Researchers pretrained 800 language models to measure how unlabeled AI-generated web text affects pretraining. After FineWeb filtering, Pangram labeled 27.5% of June 2026 web tokens as AI-generated and 31.1% by August. For data-starved models, added AI tokens first lower loss on human text, then the benefit saturates and reverses; with abundant human text, AI tokens raise loss almost immediately. Their scaling law, which reduces to Chinchilla when no AI text is present, predicts effects on models up to 3.6 times larger with 41% lower error, and they release the 83-billion-token WildAI corpus.
- After FineWeb filtering, 27.5% of June 2026 web tokens were AI-labeled, rising to 31.1% by August.
- Eight hundred models show AI tokens help data-starved training, then raise loss on human text.
- A new scaling law cuts error 41% versus prior laws for models up to 3.6 times larger.
- Authors release WildAI, an 83-billion-token labeled corpus, plus all models and code.
- Organizations
- Pangram
From the original article
1 min read at arxiv.orgWeb text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27. 5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.
By Jenna Russell · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.