titleMMDM: Measuring Massive Multitask Decision Making
authorsJunhao Shen, Qing Zong, Yue Guo, Zhanqiu Zhang, Yiwen Guo, Hong Cheng
date2026-11-10
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.14309
affiliationThe Chinese University of Hong Kong
upvotes27 (Hugging Face Daily Papers, 2026-11-13)
linksabs pdf hf code
cite
relatedOrchestra-o1: Omnimodal Agent Orchestration
Jev in the Wild: A Data-Driven Analysis of the Jev Model's Functionality, Applications and Ecosystem
PackLab: A Comprehensive Framework for Developing, Training, and Evaluating MLLMs in Robotic Bin Packing
VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
MedJev: A System One Vision-Language Decision Model for Fast and Effective Medical Image Understanding
abstractSystem-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization. Moreover, real-world decision making often involves long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together. Thus, we introduce MMDM (Massive Multitask Decision Making), a benchmark of 5,448 high-quality questions across 16 task categories, combining newly constructed tasks with source-grounded recompositions and rewrites for single-select, multi-select, and soft-label decision making in fresh contexts. It covers realistic scenarios such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs extending to 64,817 tokens. With MMDM, we conduct a comprehensive quantitative evaluation of 28 decision models, showing that many perform well on common knowledge yet struggle with interdependent options, constrained decisions, and multi-select tasks. For example, Perplexity Decider-V1.2 reaches 78.75% Hard Single-select Accuracy on world knowledge, but only 53.95% on option logic and 47.95% on constraint planning, while the best Hard Multi-select Accuracy across all models is 36.81%. These results highlight joint reasoning, constraint-aware selection, and complete multi-select recovery as key priorities for decision model development, with MMDM providing a testbed for measuring this progress.

bibtex

@misc{shen2026mmdm,
  title={MMDM: Measuring Massive Multitask Decision Making},
  author={Junhao Shen and Qing Zong and Yue Guo and Zhanqiu Zhang and Yiwen Guo and Hong Cheng},
  year={2026},
  eprint={2611.14309},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.14309}
}