Do “Evidence‐Based Policy” Clearinghouses Provide Good Advice for Local Policymakers
Bibliographic Data
| ID | 11682597 |
|---|---|
| Authors | Larry L Orr (0000-0001-7750-8580, Bloomberg School of Public Health Johns Hopkins University Baltimore Maryland USA, corresponding author) |
| Year | 2025 |
| Volume | 45 |
| Issue | 1 |
| Publication date | 2025-12-29 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Journal of Policy Analysis and Management (JOURNAL) |
| Journal identifiers | ISSN: 0276-8739 • E-ISSN: 1520-6688 |
| Publisher | Wiley (PUBLISHER • GB) |
| DOI | 10.1002/pam.70077 |
| OpenAlex | W7117475124 |
| Language | EN |
| References cited | 43 |
Policymakers are often urged to rely on “evidence‐based policy” (EBP)—adopting only interventions proven effective (i.e., positive and statistically significant in multisite impact evaluations). EBP clearinghouses chronicle and rate tests of social policy interventions. But EBP clearinghouse standards are based almost entirely on internal validity. They largely ignore whether research findings from multisite trials apply to individual localities, where much of social policy is formulated. We develop a Bayesian model of the probability that the EBP rule is sound advice to local policymakers. The model allows a direct test of the probability of a correct policy decision under the EBP rule, its positive predictive value (PPV), and its negative predictive value (NPV)—the probabilities that an intervention deemed effective by that rule will in fact be effective in a particular site (PPV), and that an intervention deemed ineffective will not be effective in a particular site (NPV), given the true impact of the intervention. These intuitive, easily calculated probabilities are major contributions of this paper. In our illustrative analysis of six multisite randomized trials, we find that under the EBP clearinghouse rule the probability of a correct policy decision, PPV, and NPV are all unacceptably low unless the cross‐site impact heterogeneity is quite low
Advice (programming · Bayesian probability · Decision rule · Intervention (counseling · Psychological intervention · Test (biology · Value (mathematics · Advanced Causal Inference Techniques · demographic modeling and climate adaptation · Gender, Labor, and Family Dynamics
Bart
Site Selection Bias in Program Evaluation
Instruments, Randomization, and Learning about Development
Evaluating replicability of laboratory experiments in economics
Making The Most Out Of Programme Evaluations and Social Experiments
Heterogeneity of variance in experimental studies
A Review of Statistical Methods for Generalizing From Evaluations of Educational Interventions
Linking program implementation and effectiveness
Wrap‐around Services Don't Improve Prisoner Reentry Outcomes
Aera Distinguished Lecture
Interpreting Effect Sizes of Education Interventions
Evidence-Based Policy
Understanding and misunderstanding randomized controlled trials
Factors relevant to the validity of experiments in social settings
| Highly cited | No |
|---|