Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal
Bibliographic Data
| ID | 6414830 |
|---|---|
| Authors | Andrew P Jaciw (0000-0002-9515-7822, Empirical Education Inc., San Mateo, CA, USA, corresponding author) |
| Year | 2024 |
| Volume | 46 |
| Issue | 1 |
| Pages | 90-127 |
| Publication date | 2024-08-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | American Journal of Evaluation (JOURNAL) |
| Journal identifiers | ISSN: 1098-2140 • E-ISSN: 1557-0878 |
| Publisher | SAGE Publishing (PUBLISHER • US) |
| DOI | 10.1177/10982140241246208 |
| OpenAlex | W4401227009 |
| Language | EN |
| References cited | 43 |
By design, randomized experiments (XPs) rule out bias from confounded selection of participants into conditions. Quasi-experiments (QEs) are often considered second-best because they do not share this benefit. However, when results from XPs are used to generalize causal impacts, the benefit from unconfounded selection into conditions may be offset by confounded selection into locations. This work shows that this tradeoff can lead to situations where estimates from QEs are less-biased from selection than are estimates from uncompromised XPs when drawing causal generalizations. This work establishes the conditions theoretically, demonstrates the idea empirically, and discusses the implications of the results
Econometrics · Generalization · Machine learning · Offset (computer science · Selection (genetic algorithm · Selection bias · Statistics · Advanced Causal Inference Techniques · Computer Science · Mathematics · Optimal Experimental Design Methods · Psychology · Statistical Methods in Clinical Trials
Designing evaluations of educational and social programs
Experimental Estimates of Education Production Functions
Empirical Benchmarks for Interpreting Effect Sizes in Research
A Summative Evaluation of RCT Methodology
Does matching overcome LaLonde's critique of nonexperimental estimators?
Misunderstandings Between Experimentalists and Observationalists about Causal Inference
The central role of the propensity score in observational studies for causal effects
Matching As An Econometric Evaluation Estimator
From Local to Global
Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs
Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact
A Within-Study Approach to Evaluating the Role of Moderators of Impact in Limiting Generalizations from “Large to Small”
The Tennessee Study of Class Size in the Early School Grades
How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment
Linking program implementation and effectiveness
Three conditions under which experiments and observational studies produce comparable causal estimates
A Conceptual Framework for Studying the Sources of Variation in Program Effects
Lurking Inferential Monsters? Quantifying Selection Bias in Evaluations of School Programs
Can Quasi‐experimental Evaluations That Rely on State Longitudinal Data Systems Replicate Experimental Results
Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions
Nonexperimental Versus Experimental Estimates of Earnings Impacts
The Effects of Small Classes on Academic Achievement
Answers and Questions about Class Size
Beyond the two disciplines of scientific psychology
| Citation velocity | historical |
|---|---|
| Highly cited | No |