Can Large Language Models Transform Computational Social Science

Bibliographic Data

ID5803658
AuthorsCaleb Ziems (Stanford University Computer Science Department. [email protected]), William A Held (Georgia Institute of Technology), William Held (Georgia Institute of Technology College of Computing. [email protected]), Omar Ahmed Shaikh (Laboratoire d'Informatique de Paris-Nord), Omar Shaikh (Stanford University Computer Science Department. [email protected]), Jiaao Chen (Georgia Institute of Technology College of Computing. [email protected]), Zhehao Zhang (0009-0006-0112-7398, Dartmouth College Department of Computer Science. [email protected]), Diyi Yang (0000-0003-1220-3983, Stanford University Computer Science Department. [email protected])
Year2023
Volume50
Issue1
Pages237-291
Publication date2023-12-12
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueComputational Linguistics (JOURNAL)
Journal identifiersISSN: 0891-2017 • E-ISSN: 1530-9312
PublisherAssociation for Computational Linguistics (PUBLISHER • US)
DOI10.1162/coli_a_00502
OpenAlexW4389636360
LanguageEN
Citations received93
References cited235

Large language models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the computational social science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that the performance of today's LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans

Bootstrapping (finance · Computational and Text Analysis Methods · Natural Language Processing Techniques · Topic Modeling