Generative Multimodal Models for Social Science

An Application with Satellite and Streetscape Imagery

Bibliographic Data

ID2331213
AuthorsTina Law (0000-0001-7631-6763, Department of Sociology, University of California, Davis, Davis, CA, USA, corresponding author), Elizabeth Roberto (0000-0001-7667-6953, Rice University)
Year2025
Volume54
Issue3
Pages889-932
Publication date2025-08-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueSociological Methods & Research (JOURNAL)
Journal identifiersISSN: 0049-1241 • E-ISSN: 1552-8294
PublisherSAGE Publications Inc (PUBLISHER)
DOI10.1177/00491241251339673
OpenAlexW4410772566
LanguageEN
Citations received3
References cited73

Although there is growing social science research examining how generative AI models can be effectively and systematically applied to text-based tasks, whether and how these models can be used to analyze images remain open questions. In this article, we introduce a framework for analyzing images with generative multimodal models, which consists of three core tasks: curation, discovery, and measurement and inference. We demonstrate this framework with an empirical application that uses OpenAI's GPT-4o model to analyze satellite and streetscape images ( n = 1,101) to identify built environment features that contribute to contemporary residential segregation in U.S. cities. We find that when GPT-4o is provided with well-defined image labels, the model labels images with high validity compared to expert labels. We conclude with thoughts for other use cases and discuss how social scientists can work collaboratively to ensure that image analysis with generative multimodal models is rigorous, reproducible, ethical, and sustainable

Generative grammar · Generative model · Geography · Remote sensing · Satellite · Satellite imagery · Artificial Intelligence · Computer Science · Geographic Information Systems Studies · Human Mobility and Location-Based Analysis · Psychology