Coding latent concepts
A human and LLM-coordinated content analysis procedure
Bibliographic Data
| ID | 21577207 |
|---|---|
| Authors | Jia Fan (0000-0001-5158-629X), Fan Jia (0000-0003-3855-532X, corresponding author), Yushi Ai, Xiaofan Liu (0000-0002-3693-5217), Yilin Deng, Yongning Li (0000-0002-3946-5951) |
| Year | 2024 |
| Volume | 41 |
| Issue | 5 |
| Pages | 324-334 |
| Publication date | 2024-10-19 |
| Open Access | No |
| Type | ARTICLE |
| Venue | Communication Research Reports (JOURNAL) |
| Journal identifiers | ISSN: 0882-4096 • E-ISSN: 1746-4099 |
| Publisher | Informa UK Limited (PUBLISHER • GB) |
| DOI | 10.1080/08824096.2024.2410263 |
| OpenAlex | W4403102389 |
| Language | EN |
| Citations received | 9 |
| References cited | 18 |
Measuring complex and latent concepts at a large-scale poses significant challenges for communication researchers. While computational and crowdsourced methods offer solutions, they often require high professional thresholds or incur significant costs. The recent advent of large language models has revolutionized content analysis. This paper employs a human and LLM-coordinated analysis procedure to measure complex and latent concepts in 1,000 public comments, exemplified by the multi-dimension concept of “deliberativeness.” We showcase the collaboration between humans and LLMs in completing complex coding tasks by designing and refining a codebook for human use and corresponding prompts for LLMs. Surprisingly, we find that fine-tuned GPT-3.5-turbo-1106 with smaller datasets can surpass GPT-4o-2024-05-13’s performance and match manual content analyses. This paper provides communication researchers with an efficient and cost-effective reference for measuring latent concepts
Content analysis · Epistemology · Social science · Sociology · Statistics · Structural equation modeling · Advanced Text Analysis Techniques · Computational and Text Analysis Methods · Psychology · Social Psychology · Topic Modeling
The Content Analysis Guidebook
Content Analysis
ChatGPT outperforms crowd workers for text-annotation tasks
A Collaborative Way of Knowing
Using Supervised Machine Learning in Automated Content Analysis
Better Crowdcoding
Content Analysis by the Crowd
Ten Steps in Scale Development and Reporting
The Validity of Sentiment Analysis
The practice of social research
Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis
Computational Grounded Theory
| Citations per year | 9 |
|---|---|
| Citation span | 2025 - 2026 (2) |
| Highly cited | No |