Identification of stone deterioration patterns with large multimodal models. Definitions and benchmarking
Bibliographic Data
| ID | 4725099 |
|---|---|
| Authors | Daniele Corradetti (0000-0001-8086-0593, Instituto Superior Técnico, corresponding author), José Delgado Rodrigues (0000-0002-5180-6130, Department of Cultural Heritage) |
| Year | 2025 |
| Volume | 71 |
| Pages | 175-183 |
| Publication date | 2025-01-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Journal of Cultural Heritage (JOURNAL) |
| Journal identifiers | ISSN: 1296-2074 • E-ISSN: 1778-3674 |
| Publisher | Elsevier BV (PUBLISHER) |
| DOI | 10.1016/j.culher.2024.11.017 |
| OpenAlex | W4405092614 |
| Language | EN |
| Citations received | 1 |
| References cited | 4 |
We define a dataset for benchmarking the capabilities of Large Multimodal Models (LMMs) in identifying and classifying stone deterioration patterns for cultural heritage conservation. • Results show LMMs have significant potential but require further fine-tuning and architectural enhancements to reliably identify patterns. • GPT-4omni achieved the highest target pattern identification rate at 42%, but no model achieved high accuracy, indicating large room for improvement. • Inconsistencies in repeated model inferences highlight the need for reproducibility and interpretability before deploying LMMs in real-life conservation workflows. The conservation of stone-based cultural heritage sites is a critical concern for preserving cultural and historical landmarks. With the advent of Large Multimodal Models, as GPT-4omni (OpenAI), Claude 3 Opus (Anthropic) and Gemini 1.5 Pro (Google), it is becoming increasingly important to define the operational capabilities of these models. In this work, we systematically evaluate the image classification capabilities of the main foundational multimodal models to recognise and categorize anomalies and deterioration patterns of stone elements that are useful in the practice of conservation and restoration of world heritage. After defining a taxonomy of the main stone deterioration patterns and anomalies, we asked the foundational models to identify a curated selection of 354 highly representative images of stone-built heritage, offering them a careful selection of labels to choose from. The result, which varies depending on the type of pattern, allowed us to identify the strengths and weaknesses of these models in the field of heritage conservation and restoration
Benchmarking · Business · Construction engineering · Forensic engineering · 3D Surveying and Cultural Heritage · Archaeological Research and Protection · Building materials and conservation · Computer Science · Engineering
| Citations per year | 1 |
|---|---|
| Citation span | 2025 - 2025 (1) |
| Highly cited | No |