Categorizing Topics Versus Inferring Attitudes
A Theory and Method for Analyzing Open-ended Survey Responses
Bibliographic Data
| ID | 12370652 |
|---|---|
| Authors | William Hobbs (0000-0002-6229-2753), Jon Green (0000-0002-6542-6745) |
| Year | 2025 |
| Volume | 33 |
| Issue | 3 |
| Pages | 231-251 |
| Publication date | 2025-01-24 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Political Analysis (JOURNAL) |
| Journal identifiers | ISSN: 1047-1987 • E-ISSN: 1476-4989 |
| Publisher | Cambridge University Press (CUP) (PUBLISHER) |
| DOI | 10.1017/pan.2024.23 |
| OpenAlex | W4406810381 |
| Language | EN |
| Citations received | 1 |
| References cited | 30 |
Past work on closed-ended survey responses demonstrates that inferring stable political attitudes requires separating signal from noise in “top of the head” answers to researchers’ questions. We outline a corresponding theory of the open-ended response, in which respondents make narrow, stand-in statements to convey more abstract, general attitudes. We then present a method designed to infer those attitudes. Our approach leverages co-variation with words used relatively frequently across respondents to infer what else they could have said without substantively changing what they meant—linking narrow themes to each other through associations with contextually prevalent words. This reflects the intuition that a respondent may use different specific statements at different points in time to convey similar meaning. We validate this approach using panel data in which respondents answer the same open-ended questions (concerning healthcare policy, most important problems, and evaluations of political parties) at multiple points in time, showing that our method’s output consistently exhibits higher within-subject correlations than hand-coding of narrow response categories, topic modeling, and large language model output. Finally, we show how large language models can be used to complement—but not, at present, substitute—our “implied word” method
Data science · Econometrics · Statistics · Computational and Text Analysis Methods · Computer Science · Electoral Systems and Political Participation · Mathematics · Psychology
The Nature and Origins of Mass Opinion
Common Knowledge
Bart
ChatGPT outperforms crowd workers for text-annotation tasks
The Construction of Social Judgments
Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale
Self-coding
Active Learning Approaches for Labeling Text
The Ideological Foundations of Affective Polarization in the U.S. Electorate
Mass Political Attitudes and the Survey Response
The Political Culture of Ambivalence
A Simple Theory of the Survey Response
Topics, Concepts, and Measurement
Constrained Citizens? Ideological Structure and Conflict Extension in the US Electorate, 1980–2016
The Open and Closed Question
Rethinking Representation
Text as Data
The Strength of Issues
Moderates
Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance
Women Also Know Stuff
Public Opinion and Public Ignorance
| Citations per year | 1 |
|---|---|
| Citation span | 2025 - 2025 (1) |
| Highly cited | No |