Expanding Access, Preserving Judgment
Large Language Models in Social Science Data Collection
Bibliographic Data
| ID | 24224993 |
|---|---|
| Authors | Chun‐Young Park (0000-0001-7847-2233, Nazarbayev University), Matthew Rains (0000-0002-0338-8691, Whitworth University) |
| Year | 2026 |
| Volume | 70 |
| Issue | 4 |
| Publication date | 2026-09-11 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | International Studies Quarterly (JOURNAL) |
| Journal identifiers | ISSN: 0020-8833 • E-ISSN: 1468-2478 |
| Publisher | Oxford University Press (OUP) (PUBLISHER) |
| DOI | 10.1093/isq/sqag084 |
| OpenAlex | W7222049794 |
| Language | EN |
| References cited | 28 |
What role should large language models (LLMs) play in social science research? The rapid rise of LLMs has generated both excitement and trepidation about their functional and ethical implications. This study evaluates their role in political science data production through the Universal Periodic Review Other Stakeholders’ Engagement (UPROSE) Dataset Project. The UPROSE project uses Universal Periodic Review reports from Non-Governmental Organizations on human rights conditions to build data on human rights and NGO engagement with international organizations. We compare LLM performance to human coders across multiple extraction tasks using established reliability metrics. LLMs perform comparably to humans for well-structured, surface-level variables but fall short in tasks requiring contextual inference or interpretive judgment. These results suggest that LLMs are best used as conditional tools that, when paired with subject-matter expertise, can expand access to high-quality data and reduce structural barriers. We conclude with best practices for hybrid workflows that enhance efficiency while maintaining interpretive quality in social science research.
Artificial Intelligence in Law · Computational and Text Analysis Methods · Ethics and Social Impacts of AI
Routledge Handbook of NGOs and International Relations
NGOs and international development
Large language models and political science
The ‘Implicit Intelligence’ of artificial intelligence. Investigating the potential of large language models in social science research
Large language models as a substitute for human experts in annotating political text
Knowledge/Power/Inequality
What Counts? How to use Different Sources of NGO Data
Using large language models to analyze political texts through natural language understanding
Codebook LLMs
Positioning Political Texts with Large Language Models by Asking and Averaging
Party Positioning Under Populist State Leaders
Global International Relations (IR) and Regional Worlds
An Examination of the Use of Large Language Models to Aid Analysis of Textual Data
| Highly cited | No |
|---|