| title | Graph Machine: Towards Better Pretraining via Edges |
|---|---|
| authors | Lintai Hou |
| date | 2026-09-02 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.02881 |
| upvotes | 5 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves O(n) complexity in its sparse layers without restricting the potentially accessible state size to O(1). Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss. |
cite
@misc{hou2026graph,
title={Graph Machine: Towards Better Pretraining via Edges},
author={Lintai Hou},
year={2026},
eprint={2609.02881},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.02881}
}