How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
A study of 800 models finds wild AI web text can hurt pretraining and proposes a new scaling law.
After FineWeb filtering, Pangram labeled 27.5% of June 2026 web tokens as AI-generated, rising to 31.1% in August. Researchers pretrained 800 language models while varying wild AI versus human tokens and fit scaling laws to held-out loss. For data-starved models, some AI text initially helps human-text loss before the benefit reverses, while models already rich in human text see loss rise almost immediately. Their new law, which separates benefit and harm and reduces to Chinchilla without AI text, predicted effects on models up to 3.6 times larger with 41% lower error. They released WildAI, an 83-billion-token labeled corpus, plus the models and code.
- 27.5% of June 2026 FineWeb tokens were AI-generated, 31.1% by August.
- Eight hundred models were pretrained across AI-to-human token ratios.
- AI tokens eventually raise loss on human text; fresh human tokens do not.
- New scaling law cut prediction error 41% versus the best prior law.
- WildAI, an 83B-token labeled corpus, models, and code were released.
- Organizations
- Pangram Labs
From the original article
1 min read at huggingface.co
Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27. 5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.
By Jenna Russell, Ben Glickenhaus, Katherine Thai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.