Doing L2 Speech Research Online
Why and How to Collect Online Ratings Data
Bibliographic Data
| ID | 21506345 |
|---|---|
| Authors | Christopher L Nagle (0000-0003-2712-2705, Iowa State University, corresponding author), Ivana Rehman (0000-0002-8590-5401, Iowa State University) |
| Year | 2021 |
| Volume | 43 |
| Issue | 4 |
| Pages | 916-939 |
| Publication date | 2021-09-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Studies in Second Language Acquisition (JOURNAL) |
| Journal identifiers | ISSN: 0272-2631 • E-ISSN: 1470-1545 |
| Publisher | Cambridge University Press (CUP) (PUBLISHER) |
| DOI | 10.1017/s0272263121000292 |
| OpenAlex | W3172685660 |
| Language | EN |
| Citations received | 14 |
| References cited | 36 |
Listener-based ratings have become a prominent means of defining second language (L2) users’ global speaking ability. In most cases, local listeners are recruited to evaluate speech samples in person. However, in many teaching and research contexts, recruiting local listeners may not be possible or advisable. The goal of this study was to hone a reliable method of recruiting listeners to evaluate L2 speech samples online through Amazon Mechanical Turk (AMT) using a blocked rating design. Three groups of listeners were recruited: local laboratory raters and two AMT groups, one inclusive of the dialects to which L2 speakers had been exposed and another inclusive of a variety of dialects. Reliability was assessed using intraclass correlation coefficients, Rasch models, and mixed-effects models. Results indicate that online ratings can be highly reliable as long as appropriate quality control measures are adopted. The method and results can guide future work with online samples
Developmental psychology · Intraclass correlation · Psychometrics · Rasch model · Rating scale · Applied Psychology · Computer Science · Linguistic Variation and Morphology · Phonetics and Phonology Research · Psychology · Speech and dialogue systems · Artificial Intelligence
Methodologies for L2 Pronunciation Research
Identification and evaluation of L1 and L2 Chinese accents
The relationship between intelligibility and comprehensibility in second language speech
The L2 Spanish Listener Ratings Dataset (L2SLRD)
Coherence and comprehensibility in second language speakers’ academic speaking performance
Automated assessment of second language comprehensibility
A comparison of lab- and web-based elicited imitation
The relationships among L2 fluency, intelligibility, comprehensibility, and accentedness
The role of multiword sequences in fluent speech
Optimizing distributed practice online
Post hoc score category collapsing for L2 pronunciation research
Exploring phonetic predictors of intelligibility, comprehensibility, and foreign accent in L2 Spanish speech
Linguistic influences on comprehensibility and accentedness in second language Korean speech
Speaking performances, stakeholder perceptions, and test scores
Disentangling accent from comprehensibility
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Emmeans
Rater Experience, Rating Scale Length, and Judgments of L2 Pronunciation
Who are the crowdworkers?
Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.
Inside the Turk
Running experiments on Amazon Mechanical Turk
Fitting Linear Mixed-Effects Models Using lme4
Using Listener Judgments to Investigate Linguistic Influences on L2 Comprehensibility and Accentedness
Can Captioned Video Benefit Second Language Pronunciation
Developing a user-oriented second language comprehensibility scale for English-medium universities
Data Collection in a Flat World
Amazon Mechanical Turk
Second Language Speakers at University
Linguistic Dimensions of Accentedness and Comprehensibility
Changing Contexts and Shifting Paradigms in Pronunciation Teaching
Motivation, Emotion, Learning Experience, and Second Language Comprehensibility Development in Classroom Settings
Effects of Second Language Pronunciation Teaching Revisited
L2 Learners' Assessments of Accentedness, Fluency, and Comprehensibility of Native and Nonnative German Speech
Foreign Accent, Comprehensibility, and Intelligibility in the Speech of Second Language Learners
The Development of L2 Oral Language Skills in Two L1 Groups
The Effect of Speaker Proficiency on Intelligibility, Comprehensibility, and Accentedness in L2 Spanish
| Unique citing works | 14 |
|---|---|
| Citations per year | 4,67 |
| Citation span | 2023 - 2026 (4) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 14 |