Graph Machine: Towards Better Pretraining via Edges
Researchers propose Graph Machine, an O(n)-state sparse architecture that replaces 75% of Qwen3-0.6B dense layers with only slight loss change.
The paper introduces the Graph Machine (GM), an architecture that maintains an O(n)-sized state accessed through sparse, dynamic routing via pointer-like edges updated differentiably by a referral mechanism resembling pointer chasing. The authors replaced 75% of dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrained from scratch on 15.7B tokens. Retrieving 2 of 4,096 tokens per KV head in each sparse layer degrades loss only slightly, while retrieving 4 marginally improves loss over the dense baseline.
- Graph Machine preserves O(n) complexity in sparse layers without capping accessible state at O(1).
- Edges act as pointer-like objects updated by a differentiable referral mechanism resembling pointer chasing.
- 75% of Qwen3-0.6B dense layers replaced with GM sparse layers; pretrained from scratch on 15.7B tokens.
- Retrieving 2 of 4,096 tokens per KV head slightly degrades loss; 4 tokens marginally improve it.
- Products
- Qwen3-0.6B
- AI models
- Qwen3-0.6BGraph Machine
From the original article
1 min read at huggingface.co
We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1).
By Lintai Hou · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.