A Note on Dropping Experimental Subjects who Fail a Manipulation Check
Bibliographic Data
| ID | 7971072 |
|---|---|
| Authors | Peter M Aronow (0000-0002-4449-0756, Yale University), Jonathon Baron (0000-0001-6576-1212, Yale University), Lauren E Pinson (0000-0001-6026-281X, Yale University), Lauren Pinson |
| Year | 2019 |
| Volume | 27 |
| Issue | 4 |
| Pages | 572-589 |
| Publication date | 2019-10-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Political Analysis (JOURNAL) |
| Journal identifiers | ISSN: 1047-1987 • E-ISSN: 1476-4989 |
| Publisher | Cambridge University Press (CUP) (PUBLISHER) |
| DOI | 10.1017/pan.2019.5 |
| OpenAlex | W3125377174 |
| Language | EN |
| Citations received | 150 |
| References cited | 17 |
Dropping subjects based on the results of a manipulation check following treatment assignment is common practice across the social sciences, presumably to restrict estimates to a subpopulation of subjects who understand the experimental prompt. We show that this practice can lead to serious bias and argue for a focus on what is revealed without discarding subjects. Generalizing results developed in Zhang and Rubin (2003) and Lee (2009) to the case of multiple treatments, we provide sharp bounds for potential outcomes among those who would pass a manipulation check regardless of treatment assignment. These bounds may have large or infinite width, implying that this inferential target is often out of reach. As an application, we replicate Press, Sagan, and Valentino (2013) with a design that does not drop subjects that failed the manipulation check and show that the findings are likely stronger than originally reported. We conclude with suggestions for practice, namely alterations to the experimental design
Arithmetic · Drop out · Econometrics · Economics · Focus (optics · Physics · Political science · Replicate · Statistics · Zhàng · Advanced Causal Inference Techniques · Computer Science · Culture, Economy, and Development Studies · Experimental Behavioral Economics Studies · Law · Mathematics · Psychology
What is the effect of providing information about the science behind vaccination? A population-based survey experiment on support for the MMR vaccine
Sector bias in public programs
Trusting on the shoulders of open giants? Open science increases trust in science for the public and academics
Can Disclaimers of Affiliation Dispel Trademark Confusion? Evidence From Two Randomized Experiments
My workgroup or my desire to serve the society? The influence of workgroup identification and public service motivation on whistleblowing intentions
Why Training Paradigms Won’t Rescue Experimental Metaethics
Hispanics’ Response to Ethnic Targeting Ads for Unhealthy Products
A Window to What We Missed
Avoiding, obliging, asserting, or integrating? A survey experiment on municipal managers’ choice of conflict management strategies
Who can serve as the proxy for public employees in public administration experiments? a cross-sample comparison
Do political and social accountability arrangements increase citizens’ legitimacy perceptions? A vignette experiment in the Netherlands
Exploring and assessing issue communication – issue communication practices and consequences of issue response strategies for organizational reputation and legitimacy
Personal Brand Equity and Telepresence's Role in Tourism Electronic Commerce Live Streaming
Leading with pride
Are the effects of servant leadership only spurious? The state of research on the causal effects of servant leadership, recommendations, and an illustrative experiment
Common methodological mistakes
Is white always the standard? Using replication to revisit and extend what we know about the leadership prototype
Do followers mind the pay gap? An experimental test of the impact of the vertical pay gap on leader effectiveness
Body-Worn Cameras and Transparency
Data Collection via Online Platforms
The disclosure paradox
The Impact of the Defense Attorney on Plea Decision-Making
In Diversity We Trust? Examining the Effect of Political Newsroom Diversity on Media Trust, Use, and Avoidance
Judging Bias
“Whatever you do, just don't let him notice you're a woman!” General beliefs on women's gender ideology as a function of topic in mixed‐gender negotiations
Public Perceptions of Child Sexual Exploitation Abroad
Behind Bars and Beyond
Belief updating in the face of misinformation
Bolstering Policy Support for Disadvantaged Groups Through Humanization
Externalities and norms in the context of a novel virus
The Impact of Attentiveness Interventions on Survey Data
Disembodied Brains
Bystanders’ perceptions on online hate speech
Beyond algorithm aversion
A straightforward trajectory strengthens support for the transition away from natural gas
Occupational gender composition is related to occupational wages
Testing the Firebreak
Populism as Dog-Whistle Politics
Attribution and Migration
Explaining the educational gradient in trust in politicians
The Moderating Role of Personality on Municipal Managers’ Conflict Management Strategies
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Barking up the wrong tree? Voter bias and female underrepresentation in Japanese politics
Public trust in advisory mini-publics
Can vaccination intentions against Covid-19 be nudged
Failing to Mobilize
Public opinion on nuclear energy and nuclear weapons
Negative Party Identification and the Use of Party Cues in the Direct Democratic Context
How gender shapes attitudes and concerns about immigration
New Questions for an Old Alliance
Elite-public gaps in support for nuclear and chemical strikes
How Dropping Subjects Who Failed Manipulation Checks Can Bias Your Results
Avoiding Post-Treatment Bias in Audit Experiments
The Generalizability of Online Experiments Conducted During the Covid-19 Pandemic
No Gender Bias in Audience Perceptions of Male and Female Experts in the News
External Threats and Public Opinion
Testing public reaction to constitutional fiscal rules violations
AI Disclosure in Strategic CSR Communication
When is an Authoritarian Leader Perceived as More Abusive
What does the public want police to do during pandemics? A national experiment
In the Light
Same Scandal, Different Standards
Forum
Can foreign aid improve the donor country's image among a third-party country's public? The case of a world heritage site restoration project
Does the prospect of further sovereignty loss fuel Euroscepticism? A population-based survey experiment
The reputational consequences of polarization for American foreign policy
Social identity and support for defunding the police in the aftermath of George Floyd’s murder
Political trust and redistribution preferences
Post-pandemic crisis communication strategy for the domestic market of Thai urban beach city
Claims of Victimhood Shield Politicians from Political Scandals
Worth the effort? Compliance costs, heuristics, and perceived program accessibility
Attention and Political Choice
Does Procedural Fairness Impact Public Perception of Judicial Opinions? Evidence from a Survey Experiment
On Reciprocity, Revenge, and Replication
The lesser evil? Experimental evidence on the strength of nuclear and chemical weapon “taboos”
Filling the EU information deficit mitigates negative EU attitudes among the least knowledgeable. Evidence from a population-based survey experiment
Blame Games in Cyberspace
Congenial messages from politicians reduce affective polarization among citizens
The impact of unelected representatives on citizens’ satisfaction with democracy
The legitimacy of land use decisions by public authorities in the Netherlands
Refugee Fatigue? How Reports of Increased Numbers of Refugees Affect Public Attitudes About Admission and Overseas Relief
Happiness Scholarship and Redistributive Preferences
How does exposure to information on ethnic discrimination inspire belief change? A preregistered population‐based survey experiment testing resonance and dissonance mechanisms
The Effectiveness of Monetary and Promotion Rewards in the Public Sector and the Moderating Effect of PSM (PSM-Reward Fit or PSM Crowding Out)
Cognitive Effects of Inattentive Responding in an MTurk Sample
A cross‐cultural replication of “citizens' blame of politicians for public service failure
A replication of “the effects of making public service employees aware of their prosocial and societal impact”
Is It Going Above and Beyond or Breaking the Rules? How Clients' Identities, Perceived Deservingness, and Outcomes Affect Citizens' Judgments of Bureaucrats' Rule Decisions
Public Budget Cuts
Supporting reparations after armed conflict
From plastic to peace
Mind over Metal
Denying the Obvious
Making inferential leaps
You can't ‘fake it till you make it’
Governing through guesstimates
Undermining Sanctuary? When Local and National Partisan Cues Diverge
You Better Shop Around
The Development of an Issue Public
Why So Secretive? Unpacking Public Attitudes toward Secrecy and Success in US Foreign Policy
The Art of Laboratory Experimentation
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Randomization Analysis of Experimental Data
Training, Wages, and Sample Selection
Attentive Turkers
Identification of Causal Effects Using Instrumental Variables
Differential Effects of Right-Wing Authoritarianism and Social Dominance Orientation on Political Candidate Support
The Effect of News Coverage Concerning the Opponents' Reaction to a Concession on Its Evaluation in the Israeli-Palestinian Conflict
Instructional manipulation checks
Power and Legitimizing Ideologies in Hierarchy‐Enhancing vs. Hierarchy‐Attenuating Environments
Emotion Priming and Attributions for Terrorism
The Messenger Overwhelming the Message
Norms, Diplomatic Alternatives, and the Social Psychology of War Support
Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys
Atomic Aversion
| Unique citing works | 150 |
|---|---|
| Citations per year | 18,75 |
| Citation span | 2018 - 2026 (9) |
| Citation velocity | current |
| Highly cited | Yes |
| Citation types | Neutral: 148 |