| title | MA-Bench: Measuring Operational Planning in Language Models |
|---|---|
| authors | Rodney Lafuente-Mercado, Robert Sincero, Phil Sage, Yuan Xue |
| date | 2026-10-13 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.19138 |
| upvotes | 5 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss κ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts. |
cite
@misc{lafuentemercado2026ma,
title={MA-Bench: Measuring Operational Planning in Language Models},
author={Rodney Lafuente-Mercado and Robert Sincero and Phil Sage and Yuan Xue},
year={2026},
eprint={2610.19138},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.19138}
}