Benchmarking AI and human text classifications in the context of newspaper frames
A multi-label LLM classification approach
Bibliographic Data
| ID | 6448504 |
|---|---|
| Authors | Alexander Tripp (0009-0007-2574-7038, Vanderbilt University, corresponding author) |
| Year | 2025 |
| Volume | 12 |
| Issue | 2 |
| Publication date | 2025-04-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Research & Politics (JOURNAL) |
| Journal identifiers | ISSN: 2053-1680 • E-ISSN: 2053-1680 |
| Publisher | SAGE Publishing (PUBLISHER • US) |
| DOI | 10.1177/20531680251332353 |
| OpenAlex | W4409466202 |
| Language | EN |
| References cited | 8 |
I examine the abilities of large language models (LLMs) to accurately classify topics related to immigration from Spanish-language newspaper articles. I benchmark various LLMs (ChatGPT and Claude) and undergraduate coders with my own codings. I prompt models to label articles with either an 8 label scheme—directly analogous to the assignment of the undergraduate coders—or a 4 label scheme—aggregating the 8 labels into broader themes. In my analyses, a Few Shot ChatGPT 4o model with 8 labels emerges as the most reliable LLM classifier and comes close to the undergraduate coders, with models using 8 labels generally outperforming their 4 label counterparts. I also find that LLMs tend toward false positive errors. This application provides practical methodological guidance for applied researchers using LLMs in data coding. Overall, I demonstrate how LLMs can be effective supplements to human coders, as well as the continued value of human coding as a benchmark for text classification
Benchmarking · Context (archaeology · Information retrieval · Media studies · Natural language processing · Newspaper · Sociology · Computer Science · History · Natural Language Processing Techniques · Text and Document Classification Technologies · Topic Modeling · Artificial Intelligence
| Highly cited | No |
|---|