| title | AutoMedBench: Towards Medical AutoResearch with Agentic AI Models |
|---|---|
| authors | Junqi Liu, Selena Song, Yuhan Wang, Jiawei Mao, Hardy Chen, Xiaoke Huang, Tianhao Qi, Pengfei Guo, Yucheng Tang, Yufan He, Can Zhao, Andriy Myronenko, Dong Yang, Daguang Xu, Yuyin Zhou, Salena Song |
| date | 2026-06-03 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.01961 |
| affiliation | University of California, Santa Cruz |
| upvotes | 21 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average. |
cite
@misc{liu2026automedbench,
title={AutoMedBench: Towards Medical AutoResearch with Agentic AI Models},
author={Junqi Liu and Selena Song and Yuhan Wang and Jiawei Mao and Hardy Chen and Xiaoke Huang and Tianhao Qi and Pengfei Guo and Yucheng Tang and Yufan He and Can Zhao and Andriy Myronenko and Dong Yang and Daguang Xu and Yuyin Zhou and Salena Song},
year={2026},
eprint={2606.01961},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.01961}
}