Quest for quality
A review of design knowledge on gamified AI training data annotation systems
Bibliographic Data
| ID | 21452720 |
|---|---|
| Authors | Simon Krohmann (Institute of Applied Informatics and Formal Description Methods, corresponding author), Sascha Rank (0000-0003-1314-2764, Institute of Applied Informatics and Formal Description Methods), Manuel Schmidt-Kraepelin (0000-0003-1280-3874, Institute of Applied Informatics and Formal Description Methods), Scott Thiebes (0000-0002-6917-1831, Tongji University), Ali Sunyaev (0000-0002-4353-8519, Technical University of Munich) |
| Year | 2026 |
| Volume | 45 |
| Issue | 10 |
| Pages | 2116-2141 |
| Publication date | 2026-06-15 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Behaviour and Information Technology (JOURNAL) |
| Journal identifiers | ISSN: 0144-929X • E-ISSN: 1362-3001 |
| Publisher | Informa UK Limited (PUBLISHER • GB) |
| DOI | 10.1080/0144929x.2025.2568932 |
| OpenAlex | W4415546651 |
| Language | EN |
| References cited | 99 |
Data annotation is a tedious, yet vital task to create AI training data. Gamification can support annotation quality by motivating and engaging annotators. Existing gamified annotation system artifacts are largely instantiations; it remains unclear how the ingrained design knowledge transcends individual and situational factors and can inform the design of gamified annotation systems as a system class. This study synthesises extant gamified annotation system artifacts and gauges the maturity of ingrained design knowledge. We conduct a semi-systematic review of 56 articles which present gamified annotation system artifacts. Beyond a broad overview of design artifacts and research activities, we derive 13 solution streams that describe design knowledge as means-end relationships between goals and gamification-based solutions. While some solution streams exhibit high maturity, the results largely confirm our initial assumption that the design knowledge base on gamified annotation systems is immature. To advance maturity, we recommend to move beyond creating new instantiations as the primary research activity and to focus on developing actionable design prescriptions. Our study contributes to creating more mature design knowledge on gamified annotation systems and offers valuable perspectives on the maturity and scholarly progression of gamification research
Annotation · Data collection · Knowledge Acquisition · Training (meteorology) · Training set · Educational Games and Gamification · Intelligent Tutoring Systems and Adaptive Learning · Online Learning and Analytics
Towards understanding the effects of individual gamification elements on intrinsic motivation and performance
Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification
The rise of motivational information systems
Toward Meaningful Engagement
The PRISMA 2020 statement
From game design elements to gamefulness
Intrinsic and extrinsic motivation from a self-determination theory perspective
Literature review as a research methodology
A web-based collaborative reading annotation system with gamification mechanisms to improve reading performance
Increasing motivation in social exercise games
Using gamification elements for competitive crowdsourcing
Analysing adaptive gamification design principles for online courses
Effects of data preprocessing on detecting autism in adults using web-based eye-tracking data
Stakeholders’ design preferences for instructional gamification
A Coefficient of Agreement for Nominal Scales
How leaderboard positions shape our motivation
| Highly cited | No |
|---|