RayOrch: Programming and Executing Lineage-Controlled Multi-Grain Dataflows for Foundation-Model Data Preparation
RayOrch keeps parent-child lineage in foundation-model data pipelines and speeds MinerU up to 15.14× on 64 GPUs.
RayOrch is a programming model and distributed engine for lineage-controlled dataflows that turn heterogeneous documents and video into foundation-model training records. Programs declare ordered, variable-cardinality expansions and matching gathers; the runtime tracks membership, parents, ordinals, and terminal states, and batches ready children across parents. Parent-scoped failures suppress only that parent's remaining work. On NVIDIA H20 GPUs it reports 15.14× speedup scaling MinerU from 4 to 64 GPUs and 7.82× for a video pipeline from 8 to 64 GPUs, with MinerU 13.1% faster than Ray Data and 29.0% faster than Daft, and Docling 16.0% faster than Ray Data.
- Preserves parent-child lineage, order, and completion across GPU batches.
- MinerU scales 15.14× from 4 to 64 NVIDIA H20 GPUs.
- A video pipeline scales 7.82× from 8 to 64 GPUs.
- Cuts MinerU time versus Ray Data and Daft, and Docling versus Ray Data.
- Code is released in the OpenDCAI/RayOrch repository.
From the original article
1 min read at huggingface.co
Preparing high quality training data for foundation models requires scalable pipelines that transform heterogeneous documents and videos into structured records. Such pipelines expand each parent item into an ordered and input dependent sequence of children, whose counts may be long tailed.
By Xiaochen Ma, Zimo Meng, Junzhu Liang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.