| title | IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications |
|---|---|
| authors | Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan |
| date | 2026-09-09 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.10539 |
| affiliation | Yale NLP Lab |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect. |
cite
@misc{ma2026ideaambig,
title={IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications},
author={Yiling Ma and Yilun Zhao and Sihong Wu and Manasi Patwardhan and Arman Cohan},
year={2026},
eprint={2609.10539},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.10539}
}