Investigating Opinions on Public Policies in Digital Media

Setting up a Supervised Machine Learning Tool for Stance Classification

Bibliographic Data

ID12971104
AuthorsChristina Viehmann (0000-0001-6673-0987, Johannes Gutenberg University Mainz, corresponding author), Tilman Beck (0000-0002-1403-8240), Marcus Maurer (0009-0000-4582-3623, Johannes Gutenberg University Mainz), Oliver Quiring (0000-0002-6671-583X, Johannes Gutenberg University Mainz), Iryna Gurevych (0000-0003-2187-7621)
Year2022
Volume17
Issue2
Pages150-184
Publication date2022-12-12
Peer ReviewedYes
Open AccessNo
TypeARTICLE
VenueCommunication Methods and Measures (JOURNAL)
Journal identifiersISSN: 1931-2458 • E-ISSN: 1931-2466
PublisherTaylor & Francis (PUBLISHER • GB)
DOI10.1080/19312458.2022.2151579
OpenAlexW4311353657
LanguageEN
Citations received3
References cited58

Supervised machine learning (SML) provides us with tools to efficiently scrutinize large corpora of communication texts. Yet, setting up such a tool involves plenty of decisions starting with the data needed for training, the selection of an algorithm, and the details of model training. We aim at establishing a firm link between communication research tasks and the corresponding state-of-the-art in natural language processing research by systematically comparing the performance of different automatic text analysis approaches. We do this for a challenging task – stance detection of opinions on policy measures to tackle the COVID-19 pandemic in Germany voiced on Twitter. Our results add evidence that pre-trained language models such as BERT outperform feature-based and other neural network approaches. Yet, the gains one can achieve differ greatly depending on the specific merits of pre-training (i.e., use of different language models). Adding to the robustness of our conclusions, we run a generalizability check with a different use case in terms of language and topic. Additionally, we illustrate how the amount and quality of training data affect model performance pointing to potential compensation effects. Based on our results, we derive important practical recommendations for setting up such SML tools to study communication texts

Artificial neural network · Feature selection · Generalizability theory · Language model · Machine learning · Natural language processing · Robustness (evolution · Task (project management · Computational and Text Analysis Methods · Computer Science · Sentiment Analysis and Opinion Mining · Topic Modeling · Artificial Intelligence

Tools

Open DOI