| title | Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists |
|---|---|
| authors | Chuhan Shi, Xiaoquan Ren, Sicheng Song, Haobo Li, Rui Sheng, Yushi Sun |
| date | 2026-07-13 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.11079 |
| affiliation | HKUST |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Existing benchmarks for scientific data analysis evaluate LLMs primarily on code execution or workflow completion, overlooking that scientific analysis serves to support distinct types of scientific claims: hypothesis exploration, statistical inference, mechanistic explanation, each with different assumptions and validity criteria. We introduce SDABench, a benchmark that reorganizes evaluation around six capabilities (descriptive, exploratory, inferential, predictive, causal, and mechanistic) across five domains (Biology, Chemistry, Environment, Geography, Physics). SDABench comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, constructed through an automated pipeline. Evaluating 15 representative LLMs, we find that models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning. SDABench further provides a five-stage error analysis framework that locates where LLMs fail: more advanced models more reliably identify the relevant scope and variables, but still struggle to select appropriate analytical procedures, model variable relationships, and draw valid conclusions. |
cite
@misc{shi2026are,
title={Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists},
author={Chuhan Shi and Xiaoquan Ren and Sicheng Song and Haobo Li and Rui Sheng and Yushi Sun},
year={2026},
eprint={2607.11079},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.11079}
}