Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions
Bibliographic Data
| ID | 11680863 |
|---|---|
| Authors | Larry L Orr (0000-0001-7750-8580, Johns Hopkins University, corresponding author), Robert B Olsen (0000-0001-5095-9058, Westat (United States)), Stephen H Bell (0000-0002-2286-5887, Westat (United States)), Ian Schmid (0000-0002-0865-7053, Johns Hopkins University), Azim Shivji (Abt Associates (Nepal)), Elizabeth A Stuart (0000-0002-9042-8611, Johns Hopkins University) |
| Year | 2019 |
| Volume | 38 |
| Issue | 4 |
| Pages | 978-1003 |
| Publication date | 2019-06-07 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Journal of Policy Analysis and Management (JOURNAL) |
| Journal identifiers | ISSN: 0276-8739 • E-ISSN: 1520-6688 |
| Publisher | Wiley (PUBLISHER • GB) |
| DOI | 10.1002/pam.22154 |
| OpenAlex | W2972019704 |
| Language | EN |
| Citations received | 4 |
| References cited | 27 |
Evidence‐based policy at the local level requires predicting the impact of an intervention to inform whether it should be adopted. Increasingly, local policymakers have access to published research evaluating the effectiveness of policy interventions from national research clearinghouses that review and disseminate evidence from program evaluations. Through these evaluations, local policymakers have a wealth of evidence describing what works, but not necessarily where. Multisite evaluations may produce unbiased estimates of the average impact of an intervention in the study sample and still produce inaccurate predictions of the impact for localities outside the sample for two reasons: (1) the impact of the intervention may vary across localities, and (2) the evaluation estimate is subject to sampling error. Unfortunately, there is relatively little evidence on how much the impacts of policy interventions vary from one locality to another and almost no evidence on the implications of this variation for the accuracy with which the local impact of adopting an intervention can be predicted using findings from an evaluation in other localities. In this paper, we present a set of methods for quantifying the accuracy of the local predictions that can be obtained using the results of multisite randomized trials and for assessing the likelihood that prediction errors will lead to errors in local policy decisions. We demonstrate these methods using three evaluations of educational interventions, providing the first empirical evidence of the ability to use multisite evaluations to predict impacts in individual localities—i.e., the ability of “evidence‐based policy” to improve local policy
Dissemination · Econometrics · Economics · Empirical evidence · Evidence-based policy · Impact evaluation · Intervention (counseling · Locality · Psychological intervention · Public economics · Randomized experiment · Sample (material · Set (abstract data type · Statistics · Advanced Causal Inference Techniques · Computer Science · Evaluation and Performance Assessment · Medicine · Psychology · School Choice and Performance
Developing effective intervention strategies to improve public participation in household waste recycling program
Hold the Bets! Should Quasi-Experiments Be Preferred to True Experiments When Causal Generalization Is the Goal
Learning from Local Government Research Partnerships in a Fragmented Political Setting
A Within-Study Approach to Evaluating the Role of Moderators of Impact in Limiting Generalizations from “Large to Small”
Empirical Benchmarks for Interpreting Effect Sizes in Research
Site Selection Bias in Program Evaluation
Linking program implementation and effectiveness
Three conditions under which experiments and observational studies produce comparable causal estimates
A Conceptual Framework for Studying the Sources of Variation in Program Effects
External Validity in Policy Evaluations That Choose Sites Purposively
Nonexperimental Versus Experimental Estimates of Earnings Impacts
| Unique citing works | 4 |
|---|---|
| Citations per year | 0,8 |
| Citation span | 2021 - 2025 (5) |
| Citation velocity | recent |
| Highly cited | No |
| Citation types | Neutral: 4 |