Multiplicity Eludes Peer Review
The Case of Covid-19 Research
Bibliographic Data
| ID | 15469259 |
|---|---|
| Authors | Oliver Gutiérrez-Hernández (0000-0003-2580-5465, Universidad de Málaga, corresponding author), Luis V García (0000-0002-5514-2941, Consejo Superior de Investigaciones Científicas) |
| Year | 2021 |
| Volume | 18 |
| Issue | 17 |
| Pages | 9304-9304 |
| Publication date | 2021-09-03 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | International Journal of Environmental Research and Public Health (JOURNAL) |
| Journal identifiers | ISSN: 1661-7827 • E-ISSN: 1660-4601 |
| Publisher | Multidisciplinary Digital Publishing Institute (PUBLISHER • CH) |
| DOI | 10.3390/ijerph18179304 |
| PMID | 34501892 |
| OpenAlex | W3197456679 |
| Language | EN |
| References cited | 30 |
Multiplicity arises when data analysis involves multiple simultaneous inferences, increasing the chance of spurious findings. It is a widespread problem frequently ignored by researchers. In this paper, we perform an exploratory analysis of the Web of Science database for COVID-19 observational studies. We examined 100 top-cited COVID-19 peer-reviewed articles based on p -values, including up to 7100 simultaneous tests, with 50% including >34 tests, and 20% > 100 tests. We found that the larger the number of tests performed, the larger the number of significant results (r = 0.87, p -6 ). The number of p -values in the abstracts was not related to the number of p -values in the papers. However, the highly significant results ( p p -6 ) with the number of p < 0.001 significances in the papers. Furthermore, the abstracts included a higher proportion of significant results (0.91 vs. 0.50), and 80% reported only significant results. Only one reviewed paper addressed multiplicity-induced type I error inflation, pointing to potentially spurious results bypassing the peer-review process. We conclude the need to pay special attention to the increased chance of false discoveries in observational studies, including non-replicated striking discoveries with a potentially large social impact. We propose some easy-to-implement measures to assess and limit the effects of multiplicity
Coronavirus disease 2019 (COVID-19 · Multiple comparisons problem · Multiplicity (mathematics · Observational study · Spurious relationship · Statistics · Type I and type II errors · COVID-19 and Mental Health · COVID-19 Clinical Research Studies · COVID-19 epidemiological studies · Mathematics · Medicine · Internal Medicine
Scientists rise up against statistical significance
The Environment and Disease
Analyzing Tables of Statistical Tests
Analyzing Tables of Statistical Tests
The ASA Statement on p -Values
An interactive web-based dashboard to track Covid-19 in real time
Controlling the False Discovery Rate
Effects of misinformation on Covid-19 individual responses and recommendations for resilience of disastrous consequences of misinformation
Placing epidemiological results in the context of multiplicity and typical correlations of exposures
Using social and behavioural science to support Covid-19 pandemic response
Antifragile
Redefine statistical significance
| Highly cited | No |
|---|