K2-Horizon open-sources six language models up to 375B-A23B
Original title: K2-Horizon: A Frontier Fleet of Open Source Language Models
The release includes stage checkpoints, pre-training data, logs, and training code, plus about 10 trillion synthetic pre-training tokens.
K2-Horizon is a fleet of six fully open language models, from a 0.9B dense model to a 375B-A23B mixture-of-experts, aimed at reasoning and agentic work. The authors introduce Mixture-of-Value Attention, softplus attention gating, and Uno lossless diffusion adapters that they say speed decoding by up to 2.2x. A 36B-A4B model using both attention changes nearly matches a 32B dense model trained on the same data with about one-seventh the active parameters. Four models share a 22T-token corpus, and artifacts are posted at ifm.ai/k2 under the LLM360 openness definition.
Unconfirmed: Benchmark leadership and the 2.2x decoding speedup are author-reported.
- Sizes run from 0.9B dense to 375B-A23B MoE.
- Uno adapters need no separate draft model.
- Pipelines produced roughly 10T synthetic pre-training tokens.
- The 36B-A4B model uses MoVA and softplus attention gating.
- Intermediate checkpoints, data, logs, and infrastructure code are released.
From the original article
1 min read at huggingface.coWe present K2-Horizon, a fleet of six fully open language models, from a 0. 9B dense model to a 375B-A23B MoE, built for general-purpose reasoning and agentic work across math and science, software engineering, deep research, office automation, and beyond. The models lead or are competitive with strong open models of their size on agentic and coding benchmarks.
By K2 Team, Zhengzhong Liu, Haonan Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.