NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
An 8.9B-parameter latent-space language model using next-concept prediction matches OLMo-3-7B pretraining loss with only 51.3% of the training tokens.
NCP-ArchPreview augments next-token prediction with Next Concept Prediction over a product-quantized concept vocabulary built from hidden states, trained jointly end-to-end. The 8.9B model was trained on 5.73T tokens from the Dolma-3 dataset, the largest latent-space language model demonstration to date. It consumes 51.3% of the tokens to reach OLMo-3-7B's final pretraining loss and outperforms it by 2.45 points on the downstream macro-average, including a 5.99-point GSM8K gain. The learned latent space also enables lightweight domain adaptation via a 17M-parameter VQ module and improves speculative drafting accepted length by 4.17%.
- Largest latent-space language model to date at 8.9B parameters trained on 5.73T Dolma-3 tokens
- Matches OLMo-3-7B final pretraining loss using only 51.3% of total training tokens
- Outperforms OLMo-3-7B by 2.45 points macro-average, with a 5.99-point gain on GSM8K
- 17M-parameter VQ module enables lightweight domain adaptation after pretraining
- Concept representations improve DFlash2 drafter accepted length by 4.17%
- Tags
- autoregressive-pretrainingdolma-3gsm8klatent-space-language-modelnext-concept-predictionolmo-3-7bquantization
- Products
- NCP-ArchPreviewOLMo-3-7BDolma-3
- AI models
- NCP-ArchPreview
From the original article
1 min read at huggingface.co
We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation.
By NCP Team, Jiaqi Cao, Chiyu Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.