MTurk Research
Review and Recommendations
Bibliographic Data
| ID | 23322698 |
|---|---|
| Authors | Herman Aguinis (0000-0002-3485-9484, Morgan State University), Isabel Villamor (0000-0003-0845-7016, George Washington University), Ravi S Ramani (0000-0003-3324-3765, Morgan State University) |
| Year | 2021 |
| Volume | 47 |
| Issue | 4 |
| Pages | 823-837 |
| Publication date | 2021-04-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Journal of Management (JOURNAL) |
| Journal identifiers | ISSN: 0149-2063 • E-ISSN: 1557-1211 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/0149206320969787 |
| Language | EN |
| Citations received | 191 |
| References cited | 75 |
The use of Amazon’s Mechanical Turk (MTurk) in management research has increased over 2,117% in recent years, from 6 papers in 2012 to 133 in 2019. Among scholars, though, there is a mixture of excitement about the practical and logistical benefits of using MTurk and skepticism about the validity of the data. Given that the practice is rapidly increasing but scholarly opinions diverge, the Journal of Management commissioned this review and consideration of best practices. We hope the recommendations provided here will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and related fields.
From Good Citizens to Bad Apples? The Effect of Boredom Proneness
The Parent Phone Use Regulation Efficacy Scale (P‐Pure)
Facial Trustworthiness Perception Across the Adult Life Span
Later Life Workplace Index
Harnessing the Potential of Older Workers Through Relationships at Work
(How Much) Do Temporal Social Comparisons Matter
Psychological inflexibility as a mediator of the relationship between adverse childhood experiences and dissociation
Stay (Close to) Humble
Examining Indirect and Direct Effects of Risk and Resilience on the Relation Between Perceived Discrimination and Eating Disorder Symptoms in Ethnic Minority American Women
Recollection of Bullshit Work Episodes (vs. Meaningful or Neutral Ones) Relates to Negative Affect, Deviant Work Behavior and Turnover Intentions
Keep Calm and Be Humble
Toward empirical process-based case conceptualization
Voting like a boss
How cultural intelligence drives expatriate performance
Motivated Reasoning at Work
Value and moral divergences between management researchers and managers
Is It Fair to be Accurate? Moral-Emotional Responses to Organizations’ AI Orientation Choices
Content creators as social influencers
The Reasonable, the Rational, and the Good
The Wise Mind Balances the Abstract and the Concrete
Physical activity among Muslim women
Examining the dualistic model of passion in addiction recovery
Integrating communication dimensions into health models
Discussing one’s fan interests with others can satisfy or threaten psychological needs
In AI we trust? Citizen perceptions of AI in government decision making
Founder ownership and system‐justifying beliefs in relation to perception toward Black Lives Matter and other social movements
Signaling Consumer Impressions with CSR Support on Social Media
The effects of transparency cues on news source credibility online
Perceived Calling Enablement
Do bilinguals get the joke? Humor comprehension in mono- and bilinguals
Societal perceptions of Covid-19 vaccine hesitancy
“You Must Try Our Taco!”
Depression and suicidality as evolved credible signals of need in social conflicts
High in the Cloud
Fentanyl Overdose History and Cocaine Use Severity Are Associated with How People Mitigate Risks of Fentanyl-Adulterated Cocaine
Electronic Cigarette Device and Liquid Characteristics
Cognitive pathways from information overload and need for cognition to information verification on social networking sites
Effects of All-Gender Workplace Facilities Signage on Adults’ Attitudes Toward Transgender and Nonbinary People and Policies
When Text is Not Enough
Too young to lead? Role incongruity explains age bias against young leaders
Knowledge hiding in the workplace
Sociosexuality is associated with disease avoidance tendencies and can decrease during a real-life disease threat
Linking job autonomy to employee engagement and exhaustion
Psychometric assessment of climate-related emotional responses
An expanded self-consistency perspective on the effects of perceived politics
Predicting employee telecommuting preferences and job outcomes amid Covid-19 pandemic
The longitudinal link between compassion toward others and subjective well-being
Determining mobile learning acceptance outside the classroom
Make No Apologies
Negative Social Exchanges During the Pandemic, Loneliness, and the Mediating Role of Stress and Feeling Misunderstood Among People at High-Risk for Covid-19 Related Complications
Digital Games for Creativity Assessment
Digital Facilitation of Group Work to Gain Predictable Performance
Proactivity and Job Search
Workaholics Anonymous? Proself Attributions of Workaholism Increase Coworker Work Effort and Badmouthing
Predicting decision-makers’ algorithm use
The curvilinear effects of relative positions in smartphone app leaderboards on physical activity
The cyborg method
The search suggestion effect (SSE)
Continuous measures of decision-difficulty captured remotely
Testing an Expanded Model of Deterrence Applied to Guilty Plea Decision Making
Disaggregating the Monolith
The influences of lecturer charismatic leadership and technology use on student online engagement, learning performance, and satisfaction
Bringing older consumers onboard to online banking
Combining egoformative and alloformative cues in a novel tabletop navigation task
Employee regret and disappointment
When and why creative performance influences job self‐efficacy
Positive biases and psychological functioning during the coronavirus disease 2019 pandemic
Just for fun
AI as a creator or a helper
Parental childhood adversity and emotional functioning
Stealing Time on the Company’s Dime
The Ethical, Societal, and Global Implications of Crowdsourcing Research
Winning the Battle but Losing the War
Considering the Dark Side of Work
Does Leader Character Have a Gender
Laughters Nurturing Tears for Leaders and Organizations
The Ripple Effect
Who Helps Who? The Role of Stigma Dimensions in Harassment Intervention
Employees Adhere More to Unethical Instructions from Human Than AI Supervisors
Employee Moral Evaluation of Supervisor Leniency for Coworkers’ Misconduct
You Don’t Care for me, So What’s the Point for me to Care for Your Business? Negative Implications of Felt Neglect by the Employer for Employee Work Meaning and Citizenship Behaviors Amid the Covid-19 Pandemic
Two studies of the perceptions of risk, benefits and likelihood of undertaking password management behaviours
Drivers of Data Quality in Advertising Research
Advertising Meat Alternatives
Consumer Responses to AI-Generated Charitable Giving Ads
The impact of source credibility and risk perception attitudes on Americans’ willingness to participate in contact tracing applications
Equivalence in international business research
Exploring user acceptance of physical intelligent assistants
A Comparison of Different Data Quality Control Methods through the Magnitude Effect in Delay Discounting
Food for thought on eating while meeting virtually
Gender-based in-group social influence can lead women to view a hostile sexist attitude as less prejudiced and more true
Identifying Core Values with a Hierarchical, Ipsative, Preference Assessment
The Effects of Leader Support for Creativity and Leader Gender on Subordinate Creative Problem‐Solving Performance
Work–life conflict and facilitation
Silent struggle
Security Theatrics
Innovative food and sustainable consumption behaviour
Make it real, make it useful! The impact of AR social experience on brand positivity and information sharing
The Hurlbert Index of Sexual Desire-Short Form
It is not only what you do, but why you do it
Conducting interactive experiments online
The Use of Online Panel Data in Management Research
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Using Nonnaive Participants Can Reduce Effect Sizes
Online panels in social science research
Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing
The Opportunities and Limitations of Using Mechanical Turk (MTURK) in Public Administration and Management Scholarship
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Measuring and Bounding Experimenter Demand
Amazon Mechanical Turk in Organizational Psychology
Systems Perspective of Amazon Mechanical Turk for Organizational Research
Nonnaïveté among Amazon Mechanical Turk workers
A reliability analysis of Mechanical Turk data
Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research
Transparency and replicability in qualitative research
Methods to detect low quality data and its implication for psychological research
Validity and Mechanical Turk
A Critical Review and Best‐Practice Recommendations for Control Variable Usage
An Inconvenient Truth
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Running experiments on Amazon Mechanical Turk
Lie for a Dime
Response Speed and Response Consistency as Mutually Validating Indicators of Data Quality in Online Samples
Survey Satisficing Inflates Reliability and Validity Measures
Methods of Detecting Insufficient Effort Responding
The viability of crowdsourcing for survey research
A Tale of Two Sample Sources
An Analysis of Data Quality
A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory
Intertemporal Differences Among MTurk Workers
Comparing Data Characteristics and Results of an Online Factorial Survey between a Population-Based and a Crowdsource-Recruited Sample
Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies
Data Collection in a Flat World
Caring about carelessness
Ascertaining the validity of individual protocols from Web-based personality inventories
Evaluating Online Labor Markets for Experimental Research
Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys
The pitfall of experimenting on the web
The impact of overtly listing eligibility requirements on MTurk
Demand Effects in Survey Experiments
The Demographic and Political Composition of Mechanical Turk Samples
On the Transportability of Laboratory Results
MTurk Character Misrepresentation
| Unique citing works | 191 |
|---|---|
| Citations per year | 38,2 |
| Citation span | 2021 - 2026 (6) |
| Citation velocity | current |
| Highly cited | Yes |
| Citation types | Neutral: 191 |