A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory
Bibliographic Data
| ID | 19600387 |
|---|---|
| Authors | Jon Sprouse (0000-0002-8875-3204, University of California, Irvine, corresponding author) |
| Year | 2010 |
| Volume | 43 |
| Issue | 1 |
| Pages | 155-167 |
| Publication date | 2010-11-25 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Behavior Research Methods (JOURNAL) |
| Journal identifiers | ISSN: 1554-351X • E-ISSN: 1554-3528 |
| Publisher | Springer Science+Business Media (PUBLISHER • DE) |
| DOI | 10.3758/s13428-010-0039-7 |
| PMID | 21287108 |
| PMCID | PMC3048456 |
| OpenAlex | W2105502623 |
| Language | EN |
| Citations received | 135 |
| References cited | 18 |
Amazon's Mechanical Turk (AMT) is a Web application that provides instant access to thousands of potential participants for survey-based psychology experiments, such as the acceptability judgment task used extensively in syntactic theory. Because AMT is a Web-based system, syntacticians may worry that the move out of the experimenter-controlled environment of the laboratory and onto the user-controlled environment of AMT could adversely affect the quality of the judgment data collected. This article reports a quantitative comparison of two identical acceptability judgment experiments, each with 176 participants (352 total): one conducted in the laboratory, and one conducted on AMT. Crucial indicators of data quality--such as participant rejection rates, statistical power, and the shape of the distributions of the judgments for each sentence type--are compared between the two samples. The results suggest that aside from slightly higher participant rejection rates, AMT data are almost indistinguishable from laboratory data
Amazon rainforest · Cognitive psychology · Data collection · Natural language processing · Sentence · Statistics · Worry · Applied Psychology · Child and Animal Learning Development · Communication · Computer Science · Engineering · Mathematics · Neurobiology of Language and Bilingualism · Psychology · Social and Intergroup Psychology · Social Psychology · Artificial Intelligence
Complexity and naturalness biases in phonotactics
Development of the Healthcare Triggering Questionnaire in adult sexual abuse survivors
SNAP judgments
Emergent positional privilege in novel English blends
Heritage languages
Testing whether the DSM-5 personality disorder trait model can be measured with a reduced set of items
Misinformation and Polarization in a High-Choice Media Environment
Selective Exposure in the Age of Social Media
Incivility on Facebook and political polarization
Amazon Mechanical Turk in Organizational Psychology
Headphone screening to facilitate web-based auditory experiments
Age-of-acquisition ratings for 30,000 English words
Data Collection in the Digital Age
Nonnaïveté among Amazon Mechanical Turk workers
MTurk Research
Inside the Turk
TurkPrime.com
The Empirical Base of Linguistics
Scalar Diversity
Knowledge wh and False Beliefs
Cognitive Heuristics, Inter-Institutional Politics, and Public Perceptions of Insulated Institutions
The Myth of Social Mobility
The effect of three basic task features on the sensitivity of acceptability judgment tasks
Linguistic Experiments and Ordinary Language Philosophy
Experimenting on Contextualism
Color adjectives, standards, and thresholds
Predicting moral judgments from causal judgments
The Importance of Place in Communicating Climate Change to Different Facets of the American Public
(No) competition between deverbal nouns and nominalized infinitives in Spanish
Frequency, acceptability, and selection
On the position of subjects in Spanish
Demonstratives as bundlers of conceptual structure
The effect of morphological structure on semantic transparency ratings
A factor analytic investigation of the Barkley deficits in executive functioning scale, short form
Examining broad intellectual abilities obtained within an mTurk internet sample
Moving experimental psychology online
Combining egoformative and alloformative cues in a novel tabletop navigation task
Challenging the presumptive link between musical preference and aggression
The affect of negativity
Physical and Mental Health Characteristics of 2,962 Adults With Subjective Cognitive Complaints
The acquisition of native language phonotactics
Picture Power? The Contribution of Visuals and Text to Partisan Selective Exposure
Worldlikeness
The Yale Grammatical Diversity Project
Perception of regional and nonnative accents
Between syntax and discourse
More evidence on the unergative–unaccusative distinction in second language grammars
Straight from the horse’s mouth
Exploring the Adaptation-mitigation Relationship
Revisiting white backlash
Perspectives of Americans and Canadians on the use and function of sex offender registries
Producing and perceiving gestures conveying height or shape
Signs of Narcissism? Reconsidering a Widely Used Measure
The Role of Engagement in Learning From Active and Incidental News Exposure on Social Media
Secular pilgrimage to places associated with writers, musicians, and visual artists
Privacy and the Panopticon
A usage-based account of subextraction effects
Gestural modality and addressee perspective influence how we reason about time
Perceived legitimacies of health‐related and motivational presenteeism and absenteeism
The role of self‐regulation in the relationship between abusive supervision and job tension
A dual‐path model of observers' responses to peer voice endorsement
The effect of online methods on epistemic inference and scalar implicature
Presidential Appointments and Public Trust
Making Judgments Based on Similarity and Proximity
The sustenance and retention of perspectival shape representations
The roles of gender and temporal distance in the recall of dissonant self-related memories
Is an off-task mind a freely-moving mind? Examining the relationship between different dimensions of thought
Implicit and explicit processes in phonological concept learning
Assessing introspective linguistic judgments quantitatively
For and against frequencies
Intuition, intonation, inconsistency, and innateness
Meta-meta-linguistics
Asymmetries in generalizing alternations to and from initial syllables
A Test of the Cognitive Assumptions of Magnitude Estimation
Five-Factor Model Personality Traits and Self-Classified Religiousness and Spirituality
Heavy NP Shift does not cause Freezing
Testing Japanese loanword devoicing
Revisiting Satiation
Cognitive sophistication does not attenuate the bias blind spot
The role of native and non-native grammars in the comprehension of possessive pronouns
Acquisition without evidence
Subliminal wh-islands in brazilian Portuguese and the consequenceS for Syntactic theory
It could turn ugly
Marking Topic or Marking Case
The viability of crowdsourcing for survey research
A Tale of Two Sample Sources
An Analysis of Data Quality
Optional Plural Agreement in Heritage Turkish Speakers’ Verb Form Choices
Data convergence in syntactic theory and the role of sentence pairs
Variation in Morphosyntax
Structural topic marking
Frequency effects in Subject Islands
Assessing the reliability of textbook data in syntax
No argument–adjunct asymmetry in reconstruction for Binding Condition C
Effects of power and individual-level cultural orientation on preferences for volunteer tourism
Using crowdsourcing to compare temporal, social temporal, and probability discounting among obese and non-obese individuals
A streamlined approach to online linguistic surveys
Imperatives under coordination
Experimental syntax and the variation of island effects in English and Italian
Prosodic end-weight reflects phrasal stress
Agreement attraction in comprehension
A New Measure of Rank Correlation
Generative linguistics within the cognitive neuroscience of language
Move Wh in a Language Without Wh Movement
Experimental Syntax
Revisiting Satiation
Gradience in linguistic data
The design and analysis of small-scale syntactic judgment experiments
Universals and grammaticality
Magnitude estimation and what it can do for your syntax
A test of the relation between working-memory capacity and syntactic island effects
Cognitive constraints and island effects
The Empirical Base of Linguistics
Magnitude Estimation of Linguistic Acceptability
| Unique citing works | 135 |
|---|---|
| Citations per year | 4,82 |
| Citation span | 1998 - 2026 (29) |
| Citation velocity | current |
| Highly cited | Yes |
| Citation types | Neutral: 131 |