Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
NVIDIA describes scaling Indic pretraining to 2.8 trillion tokens with an open data pipeline.
Stories per day · 30 days
2 RSS
Relatedclickhouse1cloudflare1compression1indic1infrastructure1nvidia1observability1open-data1pretraining1tokens1
4 stories, newest first · one card per story
Stories tagged #data-pipeline
Rewinding the exponential: How we keep hardware usage down
Cloudflare details pipeline changes that deferred about 2.5 years of hardware growth.
4
AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?
AutoDataBench finds no agent scores above 20 of 100 writing accepted training tasks in 45 minutes.
RayOrch: Programming and Executing Lineage-Controlled Multi-Grain Dataflows for Foundation-Model Data Preparation
RayOrch keeps parent-child lineage in foundation-model data pipelines and speeds MinerU up to 15.14× on 64 GPUs.