Crowdsourced Comparative Judgement for Evaluating Learner Texts

How Reliable are Judges Recruited from an Online Crowdsourcing Platform?

Bibliographic Data

ID22871285
AuthorsPeter Thwaites (0000-0003-2176-9620, Centre for English Corpus Linguistics, Institut Langage et Communication , Collège Érasme, Cardinal Mercier, 1, Louvain-la-Neuve 1348 UCLouvain), Nathan Vandeweerd (0000-0002-6498-9474, Radboud University Nijmegen), Magali Paquot (0000-0001-5687-5074, Centre for English Corpus Linguistics, Institut Langage et Communication , Collège Érasme, Cardinal Mercier, 1, Louvain-la-Neuve 1348 UCLouvain, corresponding author)
Year2025
Volume46
Issue4
Pages611-628
Publication date2025-08-28
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueApplied Linguistics (JOURNAL)
Journal identifiersISSN: 0142-6001 • E-ISSN: 1477-450X
PublisherOxford University Press (OUP) (PUBLISHER)
DOI10.1093/applin/amae048
OpenAlexW4401158115
LanguageEN
Citations received3
References cited46

Recent studies of proficiency measurement and reporting practices in applied linguists have revealed widespread use of unsatisfactory practices such as the use of proxy measures of proficiency in place of explicit tests. Learner corpus research is one specific area affected by this problem: few learner corpora contain reliable, valid evaluations of text proficiency. This has led to calls for the development of new L2 writing proficiency measures for use in research contexts. Answering this call, a recent study by Paquot et al. (2022) generated assessments of learner corpus texts using a community-driven approach in which judges, recruited from the linguistic community, conducted assessments using comparative judgement. Although the approach generated reliable assessments, its practical use is limited because linguists are not always available to contribute to data collections. This paper, therefore, explores an alternative approach, in which judges are recruited through a crowdsourcing platform. We find that assessments generated in this way can reach near identical levels of reliability and concurrent validity to those produced by members of the linguistic community.

Crowdsourcing · Inter-rater reliability · Judgement · Language proficiency · Linguistics · Mathematics education · Natural language processing · Proxy (statistics) · Rating scale · Reliability (semiconductor) · World Wide Web · Artificial Intelligence · Computer Science · Discourse Analysis in Language Studies · Interpreting and Communication in Healthcare · Natural Language Processing Techniques · Psychology

Tools

Open DOIOpen Access