OPEN-1B: A Fully Auditable Training Run
Open-1B releases a 1B-parameter model with bitwise-reproducible training, letting independent auditors verify every step of the run on commodity hardware.
The paper introduces a 'fully auditable' tier of model transparency: every training operation is reproducible with bitwise certainty on heterogeneous commodity hardware by imposing definite ordering on GPU kernel reductions, data batch ordering, and collective communication. Because replaying a full run on one machine is infeasible, a collective verification scheme lets many independent auditors certify individual steps covering the whole run. The authors release Open-1B with its full pretraining dataset, every intermediate checkpoint, the training codebase, and an audit harness. This rules out undisclosed data, injected biases, or backdoors that proof-of-learning or proof-of-training-data techniques cannot exclude.
- Bitwise-exact replay of any training step on single commodity hardware
- Deterministic ordering imposed on kernel reductions, batch ordering, collectives
- Distributed audit scheme: independent auditors each certify individual steps
- Full release: dataset, all checkpoints, code, audit harness
From the original article
1 min read at arxiv.orgOpen-source language models have a reproducibility problem. Despite releasing weights, training data, and recipes, none of them are provably reproducible due to the non-associativity of floating-point arithmetic. Deep learning frameworks often offer a deterministic execution mode, allowing reproducible operations on the same machines.
By John Donaghy · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.