| title | The Hidden Power of Scaling Factor in LoRA Optimization |
|---|---|
| authors | Zicheng Zhang, Haoran Li, Jiaxing Wang, Guoqiang Gong, Anqi Li, Yudong Hu, Ting Xiong, Yurong Gao, Junxing Hu, Zhida Jiang, Yifeng Zhang, Pengzhang Liu, Qixia Jiang |
| date | 2026-06-11 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.12883 |
| upvotes | 7 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | In Low-Rank Adaptation (LoRA), the scaling factor α is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood. In this paper, we reveal that the scaling factor α and the learning rate function differently, with α emerging as the dominant driver of effective optimization, delivering gains that cannot be replicated by learning rate scaling alone. Through the synergy of extensive empirical analysis and a theoretical Signal-Drift framework, we uncover three findings into LoRA's scaling mechanism: First, LoRA's spectral suppression smooths the optimization landscape, rendering standard hyperparameters overly conservative and creating an optimization gap. Second, when leveraging this smoothness to accelerate convergence, α outperforms the learning rate by amplifying the task signal without increasing the drift ratio. Third, the optimal scaling factor follows a sublinear relationship with the rank, well characterized by a square-root law with an unexpectedly large coefficient, revealing the insufficient scaling of existing rank-tied heuristics. Based on these insights, we propose LoRA-α, a minimalist framework that restores α to its principled regime, making LoRA compatible with standard small learning rates. Extensive evaluations across diverse tasks demonstrate that LoRA-α consistently improves performance while streamlining hyperparameter search, unleashing the learning potential of LoRA. |
cite
@misc{zhang2026hidden,
title={The Hidden Power of Scaling Factor in LoRA Optimization},
author={Zicheng Zhang and Haoran Li and Jiaxing Wang and Guoqiang Gong and Anqi Li and Yudong Hu and Ting Xiong and Yurong Gao and Junxing Hu and Zhida Jiang and Yifeng Zhang and Pengzhang Liu and Qixia Jiang},
year={2026},
eprint={2606.12883},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.12883}
}