| title | ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment |
|---|---|
| authors | Qiuyu Tian, Haojie Yin, Yingce Xia, Youyong Kong, Zequn Liu |
| date | 2026-06-04 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.00644 |
| upvotes | 1 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, and answer-generation backbones are selected to precede the task cutoffs. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Results show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. ForeSci turns forward-looking AI research judgement into a controlled benchmark for evaluating research agents as decision-making systems. |
cite
@misc{tian2026foresci,
title={ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment},
author={Qiuyu Tian and Haojie Yin and Yingce Xia and Youyong Kong and Zequn Liu},
year={2026},
eprint={2606.00644},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.00644}
}