Graph Machine: Towards Better Pretraining via Edges
Researchers propose Graph Machine, an O(n)-state sparse architecture that replaces 75% of Qwen3-0.6B dense layers with only slight loss change.
The paper introduces the Graph Machine (GM), an architecture that maintains an O(n)-sized state accessed through sparse, dynamic routing via pointer-like edges updated differentiably by a referral mechanism resembling pointer chasing. The authors replaced 75% of dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrained from scratch on 15.7B tokens. Retrieving 2 of 4,096 tokens per KV head in each sparse layer degrades loss only slightly, while retrieving 4 marginally improves loss over the dense baseline.
- Graph Machine preserves O(n) complexity in sparse layers without capping accessible state at O(1).
- Edges act as pointer-like objects updated by a differentiable referral mechanism resembling pointer chasing.
- 75% of Qwen3-0.6B dense layers replaced with GM sparse layers; pretrained from scratch on 15.7B tokens.
- Retrieving 2 of 4,096 tokens per KV head slightly degrades loss; 4 tokens marginally improve it.
Full article110 words · extracted from huggingface.co · click to collapse
We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1). Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.02881