titleSIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research
authorsJinwoo Hwang, Sukmin Cho, Tao Zhang, Soheil Abbasloo, Jongse Park, Peng Cheng
date2026-11-06
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.10329
affiliationMicrosoft Research
upvotes12 (Hugging Face Daily Papers, 2026-11-12)
linksabs pdf hf
cite
relatedStudentSim: Training LLM-based Student Simulators
SkillOpt: Executive Strategy for Self-Evolving Agent Skills
Agent World Model in the Wild: Computer-Use Agent Environment Synthesis
RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
Latent Spatial Memory for Video World Models
abstractLLM agents increasingly drive systems research and rely on simulators to evaluate candidate designs at scale. Simulators are imperfect by design: their modeling limitations can distort predictions yet leave the recommended design correct, or change which design should be selected. We introduce Siris to test whether LLMs can tell these cases apart, judging from their systems understanding when a limitation invalidates the simulator's recommendation. By perturbing a simulator, Siris constructs workload pairs with the same candidates, limitation, and simulator recommendation. The perturbation changes predictions on both workloads, yet a reference simulator supports the recommendation on only one. Evaluating thirteen frontier LLMs on LLM serving design problems, we find that identifying a limitation does not ensure a correct choice, and even the strongest models sometimes abandon valid recommendations. Self-correction through a review turn helps these models restore valid choices but introduces new errors in several others. These findings motivate evaluating systems understanding through design decisions under imperfect simulators.

bibtex

@misc{hwang2026siris,
  title={SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research},
  author={Jinwoo Hwang and Sukmin Cho and Tao Zhang and Soheil Abbasloo and Jongse Park and Peng Cheng},
  year={2026},
  eprint={2611.10329},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.10329}
}