| title | PaintBench: Deterministic Evaluation of Precise Visual Editing |
|---|---|
| authors | Kai Xu, Ellis Brown, Shrikar Madhu, Rob Fergus, He He, Saining Xie |
| date | 2026-05-29 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.00188 |
| affiliation | VISIONx @ NYU |
| upvotes | 2 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores (R^2 = 0.91, p < 0.001). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing. |
cite
@misc{xu2026paintbench,
title={PaintBench: Deterministic Evaluation of Precise Visual Editing},
author={Kai Xu and Ellis Brown and Shrikar Madhu and Rob Fergus and He He and Saining Xie},
year={2026},
eprint={2606.00188},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.00188}
}