| title | Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature |
|---|---|
| authors | Subham Ghosh, Shubham Tiwari, Mohammad Ibrahim, Abhishek Tewari |
| date | 2026-06-29 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.29667 |
| affiliation | Computational Materials Engineering Group IIT Roorkee |
| upvotes | 5 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community. |
cite
@misc{ghosh2026unlocking,
title={Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature},
author={Subham Ghosh and Shubham Tiwari and Mohammad Ibrahim and Abhishek Tewari},
year={2026},
eprint={2606.29667},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.29667}
}