Nonexperimental Versus Experimental Estimates of Earnings Impacts
Bibliographic Data
| ID | 9756344 |
|---|---|
| Authors | Steven Glazerman (0000-0002-7862-1654, Mathematica Policy Research, corresponding author), Dan Levy (0000-0003-0630-6660, Mathematica Policy Research), Dan M Levy (Mathematica Policy Research, U.S. Centers for Disease Control and Prevention), David Myers (0000-0002-2845-4093, Mathematica Policy Research) |
| Year | 2003 |
| Volume | 589 |
| Issue | 1 |
| Pages | 63-93 |
| Publication date | 2003-09-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | The Annals of the American Academy of Political and Social Science (JOURNAL) |
| Journal identifiers | ISSN: 0002-7162 • E-ISSN: 1552-3349 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/0002716203254879 |
| OpenAlex | W1967271316 |
| Language | EN |
| Citations received | 84 |
| References cited | 15 |
To assess nonexperimental (NX) evaluation methods in the context of welfare, job training, and employment services programs, the authors reexamined the results of twelve case studies intended to replicate impact estimates from an experimental evaluation by using NX methods. They found that the NX methods sometimes came close to replicating experimentally derived results but often produced estimates that differed by policy-relevant margins, which the authors interpret as estimates of bias. Although the authors identified several study design factors associated with smaller discrepancies, no combination of factors would consistently eliminate discrepancies. Even with a large number of impact estimates, the positive and negative bias estimates did not always cancel each other out. Thus, it was difficult to identify an aggregation strategy that consistently removed bias while answering a focused question about earnings impacts of a program. They conclude that although the empirical evidence from this literature can be used in the context of training and welfare programs to improve NX research designs, it cannot on its own justify the use of such designs
Actuarial science · Context (archaeology · Earnings · Econometrics · Economics · Empirical research · Estimation · Replicate · Statistics · Welfare · Accounting · Advanced Causal Inference Techniques · Computer Science · Gender, Labor, and Family Dynamics · Health Systems, Economic Evaluations, Quality of Life · Mathematics
When randomistas rule the field
Research Designs for Program Evaluation
Quasi-Experimental Designs
The Handbook of Research Synthesis and Meta-Analysis
The importance of covariate selection in controlling for selection bias in observational studies.
Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments
Standards of Evidence
Weighting Regressions by Propensity Scores
Causal Inference with Observational Data
Drawing conclusions about causes from systematic reviews of risk factors
Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science
Matching Methods for Causal Inference
Intervention Studies
Intervention Studies
An assessment of the Extractive Industries Transparency Initiative (Eiti) using the Bayesian Corruption Indicator
Supervision Following Release From Prison and Reoffending During and After the Supervision Period
Field Experiments and Natural Experiments
Evaluating Anti-Poverty Programs
How Bias Reduction Is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis
Something to Fall Back On
Do English Language Development Curriculum Materials Matter for Students’ English Proficiency
Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal
Obesity, family instability, and socioemotional health in adolescence
Do saving promotion interventions increase household savings, consumption, and investments in Sub-Saharan Africa? A systematic review and meta-analysis
Randomization, randomized trials, and analyses using observational data
The Impact of Teen Court on Rural Adolescents
Making Valid Causal Inferences About Corrective Actions by Parents From Longitudinal Data
Empirical Comparative Law
The Place of Community in Social Work Practice Research
Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact
A Within-Study Approach to Evaluating the Role of Moderators of Impact in Limiting Generalizations from “Large to Small”
Formative Evaluation
Challenges and Dilemmas in Implementing Random Assignment in Educational Research
A Comparison of Model-Based and Design-Based Impact Evaluations of Interventions in Developing Countries
The Need for Experimental Methodology in Intimate Partner Violence
Friendship Influences During Early Adolescence
What Works in Developing Nations to Get Children Into School or Keep Them There
Applying Randomized Controlled Trials and Systematic Reviews in Social Work Research
The Campbell Collaboration
How Within-Study Comparisons can Inform Research and Evaluation Design for the Third Sector
Can adult education delay retirement from the labour market
Causal Inference and Impact Evaluation
Preface – Times Have Changed
Economic Growth and Household Purchasing Power in France
Income and Wealth Inequality in France
Forty Years of Change in Labour Supply and Demand by Skill Level – Technical Progress, Labour Costs and Social Change
Fifty Years of Abstracts in the Journal Economie et Statistique
The Emergence and Consolidation of Microsimulation Methods in France
What Value Do We Attach to Climate Action
Labour Share Developments in OECD Countries Over the Past Two Decades
Can Big Push Interventions Take Small-Scale Farmers out of Poverty? Insights from the Sauri Millennium Village in Kenya
How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment
Presidential address
Do experimental and nonexperimental evaluations give different answers about the effectiveness of government-funded training programs
Three conditions under which experiments and observational studies produce comparable causal estimates
The role of random assignment in social policy research
The Internal and External Validity of the Regression Discontinuity Design
Lurking Inferential Monsters? Quantifying Selection Bias in Evaluations of School Programs
Ten years of the Workforce Investment Act (WIA)
External Validity in Policy Evaluations That Choose Sites Purposively
Internal and External Validity of the Comparative Interrupted Time‐series Design
Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions
Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison
Mitigating regulatory impact
Removing user fees for facility-based delivery services
The impact on annual earnings of adult upper secondary education in Sweden
The Cultivation of a Prosocial Value Orientation Through Community Service
The Interplay of Peer, Parent, and Adolescent Drinking
Experimental Ethnography
Experimental Evidence and Governmental Administration
Short Comparative Interrupted Time Series Using Aggregate School-Level Data in Education Research
Combining a Local Comparison Group, a Pretest Measure, and Rich Covariates
National Service and Civic Engagement
Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment
Why Propensity Scores Should Not Be Used for Matching
Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference
Causal Mechanisms
When neighborhoods matter
Stratification in college entry and completion
Field Experiments in Religion
Another Way Out
The state of educational intervention research as viewed through the lens of literacy intervention
Verdicts or Inventions
British and disengaged
Characterizing Selection Bias Using Experimental Data
Choosing among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs
Causal Effects in Nonexperimental Studies
The efficacy of psychological, educational, and behavioral treatment
Program Applicants As a Comparison Group in Evaluating Training Programs
| Unique citing works | 84 |
|---|---|
| Citations per year | 3,5 |
| Citation span | 2002 - 2026 (25) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 83 |