Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

Dados Bibliográficos

ID4030375
AutoresLucas C Coffman (0000-0002-9132-6326, Lucas C. Coffman is Assistant Professor of Economics, Ohio State University, Columbus, Ohio.), Muriel Niederle (0000-0002-4955-2186, Muriel Niederle is Professor of Economics, Stanford University, Stanford, California. Niederle is also a Research Associate, National Bureau of Economic Research, Cambridge, Massachusetts.)
Ano2015
Volume29
Fascículo3
Páginas81-98
Data de publicação2015-08-01
Peer ReviewedSim
Open AccessSim
TipoARTICLE
PeriódicoThe Journal of Economic Perspectives (JOURNAL)
Identificadores do periódicoISSN: 0895-3309 • E-ISSN: 1944-7965
EditoraAmerican Economic Association (PUBLISHER • US)
DOI10.1257/jep.29.3.81
OpenAlexW1835064865
IdiomaEN
Citações recebidas17
Referências citadas16

The social sciences-including economics-have long called for transparency in research to counter threats to producing robust and replicable results. In this paper, we discuss the pros and cons of three of the more prominent proposed approaches: pre-analysis plans, hypothesis registries, and replications. They have been primarily discussed for experimental research, both in the field including randomized control trials and the laboratory, so we focus on these areas. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Though pre-analysis plans have been lauded in the popular press and across the social sciences, we will argue that enthusiasm for pre-analysis plans should be tempered for several reasons. Hypothesis registries are a database of all projects attempted; the goal of this promising mechanism is to alleviate the "file drawer problem," which is that statistically significant results are more likely to be published, while other results are consigned to the researcher's "file drawer." Finally, we evaluate the efficacy of replications. We argue that even with modest amounts of researcher bias-either replication attempts bent on proving or disproving the published work-or modest amounts of poor replication attempts-designs that are underpowered or orthogonal to the hypothesis-replications correct even the most inaccurate beliefs within three to five replications. We offer practical proposals for how to increase the incentives for researchers to carry out replications

Economics · Enthusiasm · Incentive · Microeconomics · Research design · Social science · Sociology · Statistics · Advanced Causal Inference Techniques · Computer Science · Decision-Making and Behavioral Economics · Economic and Environmental Valuation · Mathematics · Psychology · Social Psychology