Moderating manipulation

Demystifying extremist tactics for gaming the (regulatory) system

Bibliographic Data

ID21837697
AuthorsAshley A Mattheis (0000-0002-2919-0712, Swansea University, corresponding author), Ashton Kingdon (0000-0002-0103-7361, University of Southampton)
Year2023
Volume15
Issue4
Pages478-497
Publication date2023-12-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenuePolicy & Internet (JOURNAL)
Journal identifiersISSN: 1944-2866 • E-ISSN: 1944-2866
PublisherWiley (PUBLISHER • GB)
DOI10.1002/poi3.381
OpenAlexW4389149261
LanguageEN
Citations received3
References cited32

Due to its ease of scalability and broad applicability, the use of artificial intelligence (AI) and machine learning in platform management has gained prominence. This has led to widespread debates about the use of deplatforming as the default tool for repeated or severe violations of terms or service. But technologically deterministic approaches are not infallible and can be predictable based on their actions. This opens the door for manipulation of media content and technological affordances to become tactical options for actors seeking to subvert regulation. Existing discussions often neglect the topic of manipulation of content, algorithms, or platform affordances as a primary aspect of the strategies used by extremists in relation to the difficulties of moderation from a policy perspective. This study argues that it is essential to understand how extremists and conspiracy theorists use manipulation tactics to ‘game’ the current policy, regulatory and legislative systems of content moderation. Developing approaches that attend to manipulation as a strategy and focus on platform and context‐specific tactics will generate more effective policies, platform rules, AI developments and moderation procedures. This study analyses and demystifies three primary tactics, which the authors categorize as numerology, borderlands and merchandising, regularly used by extremists online in their strategies to ‘game’ content moderation. We provide case examples from a variety of ideologies including far‐right, QAnon and male supremacism to highlight the tactics rather than ideological nature of such manipulation. We conclude with a discussion of how demystification processes could be incorporated into content moderation settings. This study contributes new insights about evasion tactics to the content moderation discussion and expands current understanding of how platforms can develop sociotechnical remedial measures

Affordance · Human–computer interaction · Ideology · Legislature · Moderation · Political science · Politics · Public relations · Sociology · Computer Science · Hate Speech and Cyberbullying Detection · Law · Misinformation and Its Impacts · Psychology · Social Media and Politics · Social Psychology · Artificial Intelligence

Tools

Open DOIOpen Access