Machine Bias. How Do Generative Language Models Answer Opinion Polls

Datos Bibliográficos

ID2331202
AutoresJulien Boelaert (0000-0001-8675-1857, CERAPS, Faculté des sciences juridiques politiques et sociales, Université de Lille, France, autor de correspondencia), Coavoux (0000-0001-7991-3555, CREST, ENSAE, Institut Polytechnique de Paris, Paris, France), E Ollion (0000-0003-3099-5240, CREST, Ecole Polytechnique, Institut Polytechnique de Paris, Paris, France), Ivaylo Petev (0000-0002-1563-655X, CREST, CNRS, Institut Polytechnique de Paris, Paris, France), Patrick Präg (0000-0001-6175-8470, CREST, ENSAE, Institut Polytechnique de Paris, Paris, France)
Año2025
Volumen54
Número3
Páginas1156-1196
Fecha de publicación2025-08-01
Peer ReviewedSí
Open AccessSí
TipoARTICLE
RevistaSociological Methods & Research (JOURNAL)
Identificadores de la revistaISSN: 0049-1241 • E-ISSN: 1552-8294
EditorialSAGE Publications Inc (PUBLISHER)
DOI10.1177/00491241251330582
OpenAlexW4409621127
IdiomaDE
Citas recibidas5
Referencias citadas36

Generative artificial intelligence (AI) is increasingly presented as a potential substitute for humans, including as research subjects. However, there is no scientific consensus on how closely these in silico clones can emulate survey respondents. While some defend the use of these 'synthetic users,' others point toward social biases in the responses provided by large language models (LLMs). In this article, we demonstrate that these critics are right to be wary of using generative AI to emulate respondents, but probably not for the right reasons. Our results show (i) that to date, models cannot replace research subjects for opinion or attitudinal research; (ii) that they display a strong bias and a low variance on each topic; and (iii) that this bias randomly varies from one topic to the next. We label this pattern 'machine bias,' a concept we define, and whose consequences for LLM-based research we further explore

Generative grammar · Linguistics · Natural language processing · Sociology · Artificial Intelligence · Computational and Text Analysis Methods · Computer Science · Hate Speech and Cyberbullying Detection · Philosophy · Psychology · Topic Modeling