Fitting z-curves to estimate the size of the Uesd file drawer and the replicability of published findings
Bibliographic Data
| ID | 13016892 |
|---|---|
| Authors | Joris Frese (0000-0002-5871-997X, European University Institute, corresponding author) |
| Year | 2024 |
| Volume | 11 |
| Issue | 3 |
| Publication date | 2024-07-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Research & Politics (JOURNAL) |
| Journal identifiers | ISSN: 2053-1680 • E-ISSN: 2053-1680 |
| Publisher | SAGE Publishing (PUBLISHER • US) |
| DOI | 10.1177/20531680241277634 |
| OpenAlex | W4401945060 |
| Language | EN |
| Citations received | 1 |
| References cited | 21 |
The unexpected event during survey design (UESD) established itself as a viable causal inference design across multiple social science disciplines in the past few years. The distribution of UESD test statistics has not yet been scrutinized for potential anomalies to the same degree as those from other causal inference methods, such as DiD, RDD, or IV. In this article, I leverage recent advances in meta-analytical methodology to estimate the replicability of statistically significant UESD findings and quantify the size of the file drawer of non-significant findings. Precisely, I aggregate 1095 ITT coefficients and standard errors from UESD studies published between 2019 and 2023 to fit their z-curve and estimate their observed discovery rate, expected discovery rate, and expected replication rate. While most statistically significant UESD findings are predicted to be replicable, the distribution of z-values also indicates publication bias toward marginally significant findings and a large file drawer of non-significant findings. The innovative z-curve methodology, which has not seen much (or any) use in political science yet, provides promising new insights beyond established tools for the assessment of publication bias, such as funnel plots or caliper tests, and can readily be applied to entire subfields of quantitative political science
Econometrics · File size · Statistics · Computer Science · Mathematics · Meta-analysis and systematic reviews · Radiomics and Machine Learning in Medical Imaging · Statistical Methods and Inference
Publication bias in the social sciences
Finite Mixture Models
Estimating the reproducibility of psychological science
The unexpected results of the peace referendum changed conflict termination preferences in Colombia
On the reliability of published findings using the regression discontinuity design in political science
Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals
Threat‐Inducing Violent Events Exacerbate Social Desirability Bias in Survey Responses
Wait and See? Public Opinion Dynamics after Terrorist Attacks
Unexpected Event during Survey Design
The file drawer problem and tolerance for null results
The Curious Case of Theresa May and the Public That Did Not Rally
Publication Bias in Empirical Sociological Research
| Citations per year | 1 |
|---|---|
| Citation span | 2025 - 2025 (1) |
| Highly cited | No |