FastKernels: Benchmarking GPU Kernel Generation in Production

Gabriele Oliaro; Yichao Fu; May Jiang; Owen Lu; Junli Wang; Zhihao Jia; Hao Zhang; Samyam Rajbhandari

title

authors

Gabriele Oliaro, Yichao Fu, May Jiang, Owen Lu, Junli Wang, Zhihao Jia, Hao Zhang, Samyam Rajbhandari

date

2026-05-22

venue

arXiv (preprint, not peer reviewed)

arXiv:2605.23215

affiliation

Snowflake

upvotes

4 (Hugging Face Daily Papers, 2026-11-01)

links

abs pdf hf

abstract

LLM-based agents for GPU kernel generation are advancing rapidly, yet their progress is fundamentally constrained by the benchmarks they optimize against. Existing benchmarks are poorly aligned with production inference frameworks: they evaluate kernels on a single GPU with synthetic inputs, ignore the surrounding compilation stack, and reward replicating known optimizations rather than discovering new ones. The resulting reward signals are misleading: agents learn to generate kernels that score well in sandboxes but introduce interface incompatibilities, compilation-stack conflicts, and silent correctness degradation when integrated into real systems. We introduce FastKernels, a kernel benchmark built around a minimal set of 46 representative architectures spanning 8 categories, whose kernels collectively subsume those of 96.2% (409/425) of HuggingFace Transformers architectures. FastKernels doubles as a minimalistic, production-grade inference framework that runs at parity with hardened systems such as vLLM and SGLang on mainstream LLM serving and substantially exceeds upstream references on under-served architectures; each task's interface mirrors the corresponding module in the state-of-the-art library for its architecture family, enabling direct deployment of optimized kernels into production codebases. Evaluating state-of-the-art kernel agents on FastKernels, we find that even the strongest agent achieves only 0.94times aggregate speedup over production baselines, with weaker agents at 0.78times and 0.53times -- confirming that benchmark-production misalignment is a critical bottleneck for the field. We release FastKernels as a stepping stone toward kernel agents whose benchmark gains translate directly into production throughput improvements. Code is available at https://github.com/Snowflake-AI-Research/fastkernels

@misc{oliaro2026fastkernels,
  title={FastKernels: Benchmarking GPU Kernel Generation in Production},
  author={Gabriele Oliaro and Yichao Fu and May Jiang and Owen Lu and Junli Wang and Zhihao Jia and Hao Zhang and Samyam Rajbhandari},
  year={2026},
  eprint={2605.23215},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.23215}
}