| title | Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms |
|---|---|
| authors | Omatharv Bharat Vaidya, Connor T. Jerzak, Nhat Ho, Chandrajit Bajaj |
| date | 2026-05-08 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.08423 |
| affiliation | Jerzak Labs |
| upvotes | 1 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | We present a data-adaptive method for parameter-efficient fine-tuning of large neural networks. Standard low-rank adaptation methods improve efficiency by restricting each layer update to a fixed low-rank form, but this static parameterization can be too rigid when the appropriate correction depends on the input and on the evolving depth-wise computation of the network. Our approach replaces a purely layer-local adapter with a shared queryable memory of low-rank update atoms. For each block of layers, the model forms a query from the current low-rank state and a running summary of previous blocks, uses this query to retrieve a content-dependent combination of shared update components via attention, and applies the resulting routed operator within the low-rank bottleneck. In this way, the method retains the efficiency and scalability of low-rank adaptation while allowing the effective update to vary across inputs and to share reusable structure across layers. The resulting architecture provides a principled middle ground between static LoRA-style updates and fully generated parameter updates: it remains compact and parameter-efficient while supporting dynamic, context-sensitive adaptation. Further, we incorporate instruction-regularization by augmenting routing logits with a language-induced prior over update atoms, thereby biasing the selection of low-rank transformations toward semantically relevant directions without generating unconstrained parameter updates. Experiments on noisy non-linear regression tasks and LLM fine-tuning suggest that this queryable update-memory formulation can improve final test performance and training stability compared to standard low-rank adaptation, while using a comparable number of trainable parameters. |
cite
@misc{vaidya2026queryable,
title={Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms},
author={Omatharv Bharat Vaidya and Connor T. Jerzak and Nhat Ho and Chandrajit Bajaj},
year={2026},
eprint={2605.08423},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.08423}
}