| title | MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks |
|---|---|
| authors | Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, Steven Hoi |
| date | 2026-08-24 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.23035 |
| affiliation | Tongyi-MAI |
| upvotes | 28 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present MobilePA-Bench, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning 13 functional domains and 212 realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: (1)~Sub-agent Collaboration---decomposing a complex task and delegating specialized work to capable sub-agents; (2)~Memory Usage---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and (3)~Skill Usage---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents. |
cite
@misc{zhu2026mobilepa,
title={MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks},
author={Yi Zhu and Xiongwei Wu and Qiyi Wang and Tingyu Qu and Jiajun Liu and Sihan Cao and Long Chen and Weigao Sun and Feida Zhu and Yiran Zhong and Steven Hoi},
year={2026},
eprint={2608.23035},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.23035}
}