Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design

Alberto Pepe; Chien-Yu Lin; Despoina Magka; Bilge Acun; Yannan Nellie Wu; Anton Protopopov; Carole-Jean Wu; Yoram Bachrach

title

authors

Alberto Pepe, Chien-Yu Lin, Despoina Magka, Bilge Acun, Yannan Nellie Wu, Anton Protopopov, Carole-Jean Wu, Yoram Bachrach

date

2026-05-15

venue

arXiv (preprint, not peer reviewed)

arXiv:2605.15871

upvotes

14 (Hugging Face Daily Papers, 2026-11-01)

links

abs pdf hf

abstract

Toward recursive self-improvement, we investigate LLM agents autonomously designing foundation models beyond standard Transformers. We introduce a dual-framework approach: AIRA-Compose for high-level architecture search, and AIRA-Design for low-level mechanistic implementation. AIRA-Compose uses 11 agents to explore fundamental computational primitives under a 24-hour budget. Agents evaluate million-parameter candidates, extrapolating top designs to 350M, 1B, and 3B scales. This yields 14 architectures across two families: AIRAformers (Transformer-based) and AIRAhybrids (Transformer-Mamba). Pre-trained at 1B scale, these consistently outperform Llama 3.2 and Composer-found baselines. On downstream tasks, AIRAformer-D and AIRAhybrid-D improve accuracy by 2.4% and 3.8% over Llama 3.2. Furthermore, AIRA-Compose finds models with highly efficient scaling frontiers: AIRAformer-C scales 54% and 71% faster than Llama 3.2 and Composer's best Transformer, while AIRAhybrid-C outscales Nemotron-2 by 23% and Composer's best hybrid by 37%. AIRA-Design tasks 20 agents with writing novel attention mechanisms for long-range dependencies and high-performing training scripts. On the Long Range Arena benchmark, agent-designed architectures reach within 2.3% and 2.6% of human state-of-the-art on document matching and text classification. On the Autoresearch benchmark, Greedy Opus 4.5 achieves 0.968 validation bits-per-byte under a fixed time budget, surpassing the published minimum. Together, these frameworks show AI agents can autonomously discover architectures and algorithmic optimizations matching or surpassing hand-designed baselines. This establishes a powerful paradigm for discovering next-generation foundation models, marking a clear step toward recursive self-improvement.

@misc{pepe2026agentic,
  title={Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design},
  author={Alberto Pepe and Chien-Yu Lin and Despoina Magka and Bilge Acun and Yannan Nellie Wu and Anton Protopopov and Carole-Jean Wu and Yoram Bachrach},
  year={2026},
  eprint={2605.15871},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.15871}
}