| title | Unlocking Feature Learning in Gated Delta Networks at Scale |
|---|---|
| authors | Yifeng Liu, Quanquan Gu |
| date | 2026-06-02 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.04048 |
| affiliation | University of California, Los Angeles |
| upvotes | 2 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization (μP) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis. |
cite
@misc{liu2026unlocking,
title={Unlocking Feature Learning in Gated Delta Networks at Scale},
author={Yifeng Liu and Quanquan Gu},
year={2026},
eprint={2606.04048},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.04048}
}