Noise Pollution
A Multi-Step Approach to Assessing the Consequences of (Not) Validating Search Terms on Automated Content Analyses
Bibliographic Data
| ID | 22010874 |
|---|---|
| Authors | Daniela Mahl (0000-0002-5330-6885, University of Zurich, corresponding author), Gerret Von Nordheim (0000-0001-7553-3838, University of Amsterdam), Lars Guenther (0000-0002-6146-164X, Universität Hamburg) |
| Year | 2023 |
| Volume | 11 |
| Issue | 2 |
| Pages | 298-320 |
| Publication date | 2023-02-07 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Digital Journalism (JOURNAL) |
| Journal identifiers | ISSN: 2167-0811 • E-ISSN: 2167-082X |
| Publisher | Informa UK Limited (PUBLISHER • GB) |
| DOI | 10.1080/21670811.2022.2114920 |
| OpenAlex | W4296999595 |
| Language | EN |
| Citations received | 8 |
| References cited | 47 |
Advances in analytical methodologies and an avalanche of digitized data have opened new avenues for (digital) journalism research—and with it, new challenges. One of these challenges concerns the sampling and evaluation of data using (non-validated) search terms in combination with automated content analyses. This challenge has largely been neglected by research, which is surprising, considering that noise slipping in during the process of data collection can generate great methodological concerns. To address this gap, we first offer a systematic interdisciplinary literature review, revealing that the validation of search terms is far from acknowledged as a required standard procedure, both in and beyond journalism research. Second, we assess the consequences of validating search terms, using a multi-step approach and investigating common research topics from the field of (digital) journalism research. Our findings show that careless application of non-validated search terms has its pitfalls: while scattershot search terms can make sense in initial data exploration, final inferences based on insufficiently validated search terms are at higher risk of being obscured by noise. Consequently, we provide a step-by-step recommendation for developing and validating search terms
Data collection · Data mining · Data science · Information retrieval · Journalism · Management science · Sociology · Climate Change Communication and Perception · Computational and Text Analysis Methods · Computer Science · Data Analysis with R · Mathematics · Artificial Intelligence
Ethical challenges in contemporary quantitative content analysis
Understanding climate-related visual storytelling on TikTok
Measuring racism and related concepts using computational text-as-data approaches
Topic modeling three decades of climate change news in Denmark
A Distant Threat? The Framing of Climate Futures Across Four Countries
WordPPR
Veiled conspiracism
Challenges of and approaches to data collection across platforms and time
Making Artificial Intelligence Work for Investigative Journalism
Quanteda
Validation of Database Search Terms for Content Analysis
Taking Stock of the Toolkit
Issues and Best Practices in Content Analysis
STM
Der „Computational Turn“
Better off without You? How the British Media Portrayed EU Citizens in Brexit News
Applying LDA Topic Modeling in Communication Research
Expert-Informed Topic Models for Document Set Discovery
Overcoming Language Barriers
Assessing the Reporting of Reliability in Published Content Analyses
Three Gaps in Computational Text Analysis Methods for Social Sciences
Public microblogging on climate change
Reporting on climate change
Polarized frames on “climate change” and “global warming” across countries and states
Compounds, creativity and complexity in climate change communication
Climate change in news media across the globe
Distilling Issue Cycles From Large Databases
Computer‐Assisted Keyword and Document Set Discovery from Unstructured Text
If You Have Choices, Why Not Choose (and Share) All of Them? A Multiverse Approach to Understanding News Engagement on Social Media
Searching for online news content
In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis
No Longer Lost in Translation
Automated Text Classification of News Articles
Text Preprocessing For Unsupervised Learning
Computer-Assisted Text Analysis for Comparative Politics
Large-Scale Computerized Text Analysis in Political Science
External Validity
Critical Questions for Big Data
Adapting computational text analysis to social science (and vice versa)
| Unique citing works | 8 |
|---|---|
| Citations per year | 2,67 |
| Citation span | 2023 - 2025 (3) |
| Citation velocity | recent |
| Highly cited | No |
| Citation types | Neutral: 6 |