Does F1 fool you? A survey on annotator metadata transparency in hate speech detection
Bibliographic Data
| ID | 8246147 |
|---|---|
| Authors | Laura Costa Ribeiro (Universidade Federal Fluminense), Aline Paes (0000-0002-9089-7303, Universidade Federal Fluminense) |
| Year | 2026 |
| Pages | 1-21 |
| Publication date | 2026-03-18 |
| Open Access | No |
| Type | ARTICLE |
| Venue | Journal of Information Communication and Ethics in Society (JOURNAL) |
| Journal identifiers | ISSN: 1758-8871 • E-ISSN: 1477-996X |
| Publisher | Emerald (PUBLISHER) |
| DOI | 10.1108/jices-10-2025-0298 |
| OpenAlex | W7137196976 |
| Language | EN |
| References cited | 23 |
Purpose This study aims to discuss transparency and accountability in hate speech detection and adjacent fields, with a focus on misogyny, by analyzing annotator documentation; as well as to re-examine the prevailing overreliance on performance metrics alone for model evaluation and to emphasize the critical role of annotator documentation in ensuring equitable and trustworthy detection systems. Design/methodology/approach The discussion is based on the analysis of annotator documentation across 25 studies and a post hoc framework of six disclosure dimensions: annotator selection, quantity, training, demographics, compensation and expertise. Building on this framework, this paper introduces a novel weighted annotator metadata transparency (AMT) score that prioritizes demographics and domain expertise as critical factors for equitable model development. Findings The results of this study reveals significant transparency gaps: surveyed articles achieved only a 27% average AMT score, with particularly concerning deficiencies in demographics (20%) and domain expertise (0%). Furthermore, this paper identifies an inverse relationship between transparency and performance: models with higher F1 scores consistently demonstrated poorer annotator documentation, while more transparent studies clustered in the lower performance range. Originality/value This study introduces a novel, weighted framework (the AMT score) for evaluating AMT. The finding of an inverse relationship between reported model performance and transparency is a critical and original contribution that challenges current evaluation paradigms. The authors conclude by providing a set of best practices to guide the research community toward more equitable and trustworthy model development
Comparability · Demographics · Documentation · Metadata · Proxy (statistics · Set (abstract data type · Transparency (behavior · Trustworthiness · Emotion and Mood Recognition · Hate Speech and Cyberbullying Detection · Spam and Phishing Detection
| Highly cited | No |
|---|