| title | Weight-Space Programming Doesn't Need Transformers |
|---|---|
| authors | Hyunjong Ok, Jaeho Lee |
| date | 2026-11-17 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2611.22055 |
| affiliation | Pohang University of Science and Technology |
| upvotes | 49 (Hugging Face Daily Papers, 2026-11-19) |
| links | abs pdf hf code |
| cite | |
| related | Taming VLAs under Robot Execution Errors: Self-Compensation and Stress Testing Hi-Q: Hierarchical Evidence-guided Query Refinement for Multi-Hop Question Answering Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models When to Switch: Reliable Action-Chunk Extension for Vision-Language-Action Models CANOPY: Adaptive-Granularity Evidence Compression for Multimodal RAG |
| abstract | Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. From this, we propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer. We also test pretrained weights, other benchmarks, and tasks where models generate answers. Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights. |
bibtex
@misc{ok2026weight,
title={Weight-Space Programming Doesn't Need Transformers},
author={Hyunjong Ok and Jaeho Lee},
year={2026},
eprint={2611.22055},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2611.22055}
}