titleGeneralization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs
authorsNagham Omar, Mahmoud Jabarin, Maya Rozenshtein, Rom Himelstein, Avi Mendelson, Amit LeVi
date2026-10-01
venuearXiv (preprint, not peer reviewed)
idarXiv:2610.01428
affiliationTechnion Israel institute of technology
upvotes7 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractGeneralization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed in different ways. Existing work typically evaluates generalization through aggregate accuracy on a single prompt format, task, or set of variations, which conflates robustness with overall benchmark performance. In this work, we show generalization evaluation at the level of individual examples, across multiple input variants, and across different aspects of model behavior, focusing on variability rather than reducing performance to a score that can be improved through narrow training or other ways that obfuscate generalization evaluation. Following this view, we introduce the Stability-Aware Generalization Objective (SAGO), a framework that measures how much model behavior changes for the same input under different variations and benchmarks, capturing variability across several dimensions including generation consistency, internal activations, confidence, and response mirroring. We show that many commonly used models exhibit statistically significant and consistent generalization instability: no model generalizes uniformly, behavioral axes capture independent failure modes, and cross-dataset variation can reverse model rankings.

cite

@misc{omar2026generalization,
  title={Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs},
  author={Nagham Omar and Mahmoud Jabarin and Maya Rozenshtein and Rom Himelstein and Avi Mendelson and Amit LeVi},
  year={2026},
  eprint={2610.01428},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.01428}
}