Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

Datos Bibliográficos

ID4030375
AutoresLucas C Coffman (0000-0002-9132-6326, Lucas C. Coffman is Assistant Professor of Economics, Ohio State University, Columbus, Ohio.), Muriel Niederle (0000-0002-4955-2186, Muriel Niederle is Professor of Economics, Stanford University, Stanford, California. Niederle is also a Research Associate, National Bureau of Economic Research, Cambridge, Massachusetts.)
Año2015
Volumen29
Número3
Páginas81-98
Fecha de publicación2015-08-01
Peer ReviewedSí
Open AccessSí
TipoARTICLE
RevistaThe Journal of Economic Perspectives (JOURNAL)
Identificadores de la revistaISSN: 0895-3309 • E-ISSN: 1944-7965
EditorialAmerican Economic Association (PUBLISHER • US)
DOI10.1257/jep.29.3.81
OpenAlexW1835064865
IdiomaEN
Citas recibidas17
Referencias citadas16

The social sciences-including economics-have long called for transparency in research to counter threats to producing robust and replicable results. In this paper, we discuss the pros and cons of three of the more prominent proposed approaches: pre-analysis plans, hypothesis registries, and replications. They have been primarily discussed for experimental research, both in the field including randomized control trials and the laboratory, so we focus on these areas. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Though pre-analysis plans have been lauded in the popular press and across the social sciences, we will argue that enthusiasm for pre-analysis plans should be tempered for several reasons. Hypothesis registries are a database of all projects attempted; the goal of this promising mechanism is to alleviate the "file drawer problem," which is that statistically significant results are more likely to be published, while other results are consigned to the researcher's "file drawer." Finally, we evaluate the efficacy of replications. We argue that even with modest amounts of researcher bias-either replication attempts bent on proving or disproving the published work-or modest amounts of poor replication attempts-designs that are underpowered or orthogonal to the hypothesis-replications correct even the most inaccurate beliefs within three to five replications. We offer practical proposals for how to increase the incentives for researchers to carry out replications

Economics · Enthusiasm · Incentive · Microeconomics · Research design · Social science · Sociology · Statistics · Advanced Causal Inference Techniques · Computer Science · Decision-Making and Behavioral Economics · Economic and Environmental Valuation · Mathematics · Psychology · Social Psychology