titleOVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation
authorsKaiyu Li, Zepeng Xin, Zixuan Jiang, Jing Fu, Lanxuan Xue, Lingyu Zhang, Xiangyong Cao
date2026-07-29
venuearXiv (preprint, not peer reviewed)
idarXiv:2607.27278
affiliationearth-insights
upvotes14 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractOpen-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms. To fill this gap, we introduce OVEarth-Bench, which extends existing evaluation in two directions: category breadth, through broad hierarchical category coverage with positive and negative expressions, and query diversity, through vocabulary, referring, and reasoning queries. The benchmark supports mask and box localization under a unified zero-shot protocol. We evaluate a broad set of general and EO-specific methods. The evaluation reveals that: (1) the performance of current methods remains limited, while broader category coverage yields more stable model rankings; (2) MLLM-based methods achieve the strongest overall performance; and (3) EO-specific methods generally underperform general models and rarely match the strongest methods. These findings provide guidance for future open-vocabulary EO method design and highlight the importance of developing more realistic, diverse, high-quality, and large-scale benchmarks for reliable evaluation. Our data and evaluation package are released at https://earth-insights.github.io/OVEarth-bench.

cite

@misc{li2026ovearth,
  title={OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation},
  author={Kaiyu Li and Zepeng Xin and Zixuan Jiang and Jing Fu and Lanxuan Xue and Lingyu Zhang and Xiangyong Cao},
  year={2026},
  eprint={2607.27278},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2607.27278}
}