The fabrics of machine moderation

Studying the technical, normative, and organizational structure of Perspective API

Bibliographic Data

ID5260469
AuthorsBernhard Rieder (0000-0002-2404-9277, University of Amsterdam, corresponding author), Yarden Skop (0000-0001-7833-9811, University of Siegen)
Year2021
Volume8
Issue2
Publication date2021-07-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueBig Data & Society (JOURNAL)
Journal identifiersISSN: 2053-9517 • E-ISSN: 2053-9517
PublisherSAGE Publications Inc (PUBLISHER)
DOI10.1177/20539517211046181
OpenAlexW3206808617
LanguageEN
Citations received27
References cited34

Over recent years, the stakes and complexity of online content moderation have been steadily raised, swelling from concerns about personal conflict in smaller communities to worries about effects on public life and democracy. Because of the massive growth in online expressions, automated tools based on machine learning are increasingly used to moderate speech. While 'design-based governance' through complex algorithmic techniques has come under intense scrutiny, critical research covering algorithmic content moderation is still rare. To add to our understanding of concrete instances of machine moderation, this article examines Perspective API, a system for the automated detection of 'toxicity' developed and run by the Google unit Jigsaw that can be used by websites to help moderate their forums and comment sections. The article proceeds in four steps. First, we present our methodological strategy and the empirical materials we were able to draw on, including interviews, documentation, and GitHub repositories. We then summarize our findings along five axes to identify the various threads Perspective API brings together to deliver a working product. The third section discusses two conflicting organizational logics within the project, paying attention to both critique and what can be learned from the specific case at hand. We conclude by arguing that the opposition between 'human' and 'machine' in speech moderation obscures the many ways these two come together in concrete systems, and suggest that the way forward requires proactive engagement with the design of technologies as well as the institutions they are embedded in

Documentation · Knowledge management · Machine learning · Moderation · Normative · Political science · Politics · Public relations · Scrutiny · Sociology · Adversarial Robustness in Machine Learning · Computer Science · Ethics and Social Impacts of AI · Hate Speech and Cyberbullying Detection · Law · Artificial Intelligence

Tools

Open DOIOpen Access