Using Mechanical Turk to Obtain and Analyze English Acceptability Judgments
Bibliographic Data
| ID | 4789786 |
|---|---|
| Authors | Edward Gibson (0000-0002-5912-883X, Institute of Cognitive and Brain Sciences), Steve Piantadosi (Institute of Cognitive and Brain Sciences), Kristina Fedorenko (Smith College) |
| Year | 2011 |
| Volume | 5 |
| Issue | 8 |
| Pages | 509-524 |
| Publication date | 2011-08-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Language and Linguistics Compass (JOURNAL) |
| Journal identifiers | ISSN: 1749-818X • E-ISSN: 1749-818X |
| Publisher | Wiley (PUBLISHER • GB) |
| DOI | 10.1111/j.1749-818x.2011.00295.x |
| OpenAlex | W2127452865 |
| Language | EN |
| Citations received | 60 |
| References cited | 30 |
The prevalent method in theoretical syntax and semantics research involves obtaining a judgment of the acceptability of a sentence/meaning pair, typically by just the author of the paper, sometimes with feedback from colleagues. The weakness of the traditional non-quantitative single-sentence/single-participant methodology, along with the existence of cognitive and social biases, has the unwanted effect that claims in the syntax and semantics literature cannot be trusted. Even if most of the judgments in an arbitrary syntax/semantics paper can be substantiated with rigorous quantitative experiments, the existence of a small set of judgments that do not conform to the authors' intuitions can have a large effect on the potential theories. Whereas it is clearly desirable to quantitatively evaluate all syntactic and semantic hypotheses, it has been time-consuming in the past to find a large pool of naïve experimental participants for behavioral experiments. The advent of Amazon.com's Mechanical Turk now makes this process very simple. Mechanical Turk is a marketplace interface that can be used for collecting behavioral data over the internet quickly and inexpensively. The cost of using an interface like Mechanical Turk is minimal, and the time that it takes for the results to be returned is very short. Many linguistic surveys can be completed within a day, at a cost of less than $50. In this paper, we provide detailed instructions for how to use our freely available software in order to (a) post-linguistic acceptability surveys to Mechanical Turk; and (b) extract and analyze the resulting data
Linguistics · Natural language processing · Programming language · Sentence · Syntax · Computer Science · Language, Discourse, Communication Strategies · Neurobiology of Language and Bilingualism · Psychology · Syntax, Semantics, Linguistic Variation · Artificial Intelligence
Investigating the distribution of some (but not all) implicatures using corpora and web-based methods
Marking Topic or Marking Case
SNAP judgments
Constraints on Donkey Pronouns
Heritage languages
Zipf’s word frequency law in natural language
Headphone screening to facilitate web-based auditory experiments
Age-of-acquisition ratings for 30,000 English words
The Present Tense is not Vacuous
A Pragmatic Account of Complexity in Definite Antecedent-Contained-Deletion Relative Clauses
When the present lies in the past
English middles and implicit arguments
Morphological generalization in bilingual language production
Do older adults construct more emotionally gratifying social environments than younger adults? Evidence from a social network decision task
The affect of negativity
First and second language speakers’ sensitivity to the distributional properties of wh -clauses
Extraction from subjects
Neural evidence suggests phonological acceptability judgments reflect similarity, not constraint evaluation
A noisy-channel approach to depth-charge illusions
Discourse-based constraints on long-distance dependencies generalize across constructions in English and French
The Yale Grammatical Diversity Project
Using two-alternative forced choice tasks and Thurstone’s law of comparative judgments for code-switching research
Between syntax and discourse
Straight from the horse’s mouth
Emotional Crisis Communication
A usage-based account of subextraction effects
How much risk can you stomach? Individual differences in the tolerance of perceived risk across gender and risk domain
The effect of online methods on epistemic inference and scalar implicature
Testing alternative theoretical accounts of code-switching
El book or the libro ? Insights from acceptability judgments into determiner/noun code-switches
A quantitative investigation of the imperative-and-declarative construction in English
Heavy NP Shift does not cause Freezing
The interpretation of syntactically unconstrained anaphors in Turkish heritage speakers
The role of native and non-native grammars in the comprehension of possessive pronouns
Acquisition without evidence
Optional Plural Agreement in Heritage Turkish Speakers’ Verb Form Choices
Corpus linguistics in language testing research
Frequency effects in Subject Islands
Assessing the reliability of textbook data in syntax
No argument–adjunct asymmetry in reconstruction for Binding Condition C
Selectional Violations in Coordination
Passive do so
A streamlined approach to online linguistic surveys
Prosodic end-weight reflects phrasal stress
Gestural agreement
Packaging Information as Fact Versus Opinion
Prosody and the Interpretation of Hierarchically Ambiguous Discourse
Do They Really Mean It? Children’s Inference of Speaker Intentions and the Role of Age and Gender
Quantifying geographical variation in acceptability judgments in regional American English dialect syntax
Microvariation in the have yet to construction
The Southern Dative Presentative Meets Mechanical Turk
Extraction from Present Participle Adjuncts
CPs Move Rightward, Not Leftward
A judgment study of word-length preferences in Chinese NN compounds
Using audio stimuli in acceptability judgment experiments
Effects of Processing on the Acceptability of "Frozen" Extraposed Constituents
A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010
Why we need a gradient approach to word order
Prominence and coherence in a Bayesian theory of pronoun interpretation
Rises inform, and plateaus remind
Analyzing Linguistic Data
Data Analysis Using Regression and Multilevel/Hierarchical Models
Movement in Language
Phrasal Movement and Its Kin
Core Syntax
On the Failure to Eliminate Hypotheses in a Conceptual Task
Confirmation Bias
Data in generative grammar
Generative linguistics within the cognitive neuroscience of language
Psycholinguistics, formal grammars, and cognitive science
A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory
Cross-linguistic Variation in a Processing Account
Experimental Syntax
The Empirical Base of Linguistics
Syntactic Judgment Experiments
Amnestying Superiority Violations
Adding a Third Wh-phrase Does Not Increase the Acceptability of Object-initial Multiple-wh-questions
The design and analysis of small-scale syntactic judgment experiments
Intuitions in linguistic argumentation
Magnitude estimation and what it can do for your syntax
On the Informativity of Different Measures of Linguistic Acceptability
Magnitude Estimation of Linguistic Acceptability
| Unique citing works | 60 |
|---|---|
| Citations per year | 4 |
| Citation span | 2011 - 2025 (15) |
| Citation velocity | recent |
| Highly cited | No |
| Citation types | Neutral: 57 |