Beyond sentiment

An algorithmic strategy for identifying evaluations within large text corpora

Bibliographic Data

ID12971016
AuthorsMaximilian Overbeck (0000-0003-3658-5584, Hebrew University of Jerusalem, corresponding author), Christian Baden (0000-0002-3771-3413, Hebrew University of Jerusalem), Tali Aharoni (0000-0002-2138-8329, Hebrew University of Jerusalem), Eedan R Amit-Danhi (0000-0002-7029-218X, Hebrew University of Jerusalem), K Tenenboim-Weinblatt (0000-0001-9268-3969, Hebrew University of Jerusalem)
Year2023
Volume19
Issue1
Pages24-45
Publication date2023-12-07
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueCommunication Methods and Measures (JOURNAL)
Journal identifiersISSN: 1931-2458 • E-ISSN: 1931-2466
PublisherTaylor & Francis (PUBLISHER • GB)
DOI10.1080/19312458.2023.2285783
PMID40012723
OpenAlexW3183640194
LanguageEN
Citations received3
References cited58

In this paper, we propose a new strategy for classifying evaluations in large text corpora, using supervised machine learning (SML). Departing from a conceptual and methodological critique of the use of sentiment measures to recognize object-specific evaluations, we argue that a key challenge consists in determining whether a semantic relationship exists between evaluative expressions and evaluated objects. Regarding sentiment terms as merely potentially evaluative expressions, we thus use a SML classifier to decide whether recognized terms have an evaluative function in relation to the evaluated object. We train and test our classifier on a corpus of 10,004 segments of election coverage from 16 major U.S. news outlets and Tweets by 10 prominent U.S. politicians and journalists. Specifically, we focus on evaluations of political predictions about the outcomes and implications of the 2016 and 2020 U.S. presidential elections. We show that our classifier consistently outperforms both off-the-shelf sentiment tools and a pre-trained transformer-based sentiment classifier. Critically, our classifier correctly discards numerous non-evaluative uses of common sentiment terms, whose inclusion in conventional analyses generates large amounts of false positives. We discuss contributions of our approach to the measurement of object-specific evaluations and highlight challenges for future research

Classifier (UML · False positive paradox · Machine learning · Natural language processing · Sentiment analysis · Computational and Text Analysis Methods · Computer Science · Sentiment Analysis and Opinion Mining · Topic Modeling · Artificial Intelligence

Tools

Open DOIOpen Access