GPT models for text annotation
An empirical exploration in public policy research
Bibliographic Data
| ID | 6330575 |
|---|---|
| Authors | Alexander Churchill (Albers School of Business and Economics Seattle University Seattle Washington USA), Shamitha Pichika (Department of Computer Science Seattle University Seattle Washington USA), Chengxin Xu (0000-0003-4930-9028, School of Public Affairs and Nonprofit Leadership Seattle University Seattle Washington USA), Ying Liu (0000-0001-5584-7805, School of Public Affairs and Administration Rutgers University Newark New Jersey USA) |
| Year | 2025 |
| Publication date | 2025-05-02 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Policy Studies Journal (JOURNAL) |
| Journal identifiers | ISSN: 0190-292X • E-ISSN: 1541-0072 |
| Publisher | Wiley (PUBLISHER • GB) |
| DOI | 10.1111/psj.70034 |
| OpenAlex | W4410035949 |
| Language | EN |
| Citations received | 6 |
| References cited | 30 |
Text annotation, the practice of labeling text following a predetermined scheme, is essential to qualitative public policy research. Despite its importance, annotating large qualitative data faces challenges of high labor and time costs. Recent developments in large language models (LLMs), specifically models with generative pretrained transformers (GPTs), show a potential approach that may alleviate the burden of manual text annotation. In this report, we first introduce a small sample pretest strategy for researchers to decide whether to use Open AI's GPT models for text annotation. In addition, we test if GPT models can substitute human coders by comparing the results of two GPT models with different prompting strategies against human annotation. Using email messages collected from a national corresponding experiment in the US nursing home market as an example, on average, we demonstrate 86.25% percentage agreement between GPT and human annotations. We also show that GPT models possess context‐based limitations. Our report ends with reflections and suggestions for readers who are interested in using GPT models for text annotation
Annotation · Empirical research · Epistemology · Political science · Public policy · Regional science · Sociology · Computational and Text Analysis Methods · Computer Science · Philosophy · Public Administration · Topic Modeling · Artificial Intelligence
Combining Rules and Capacity
Running With Scissors? Integrating GPT Models Into Public Policy Research
The Stories Individuals “Like”
Re‐Imagining the Epistemic Possibilities of GPT for Public Administration Research in Competitive Settings
The Epistemic Impact of Large Language Models on Policymaking
Coalition stability and cohesion in a transgender policy conflict
Persistent Anti-Muslim Bias in Large Language Models
Bootstrap confidence intervals
Using cognitive psychology to understand GPT-3
ChatGPT outperforms crowd workers for text-annotation tasks
Thematic analysis.
The Language of Creativity
How qualitative research methods can be leveraged to strengthen mixed methods research in public policy and public administration
No country for model minorities
Robotic Bureaucracy
Institutional Analysis with the Institutional Grammar
Collaborating with the Machines
Machine Learning for Public Administration Research, With Application to Organizational Reputation
| Unique citing works | 6 |
|---|---|
| Citations per year | 6 |
| Citation span | 2025 - 2026 (2) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 6 |