Correcting for Bias in Psychology
A Comparison of Meta-Analytic Methods
Bibliographic Data
| ID | 23325992 |
|---|---|
| Authors | Evan C Carter (DEVCOM Army Research Laboratory, corresponding author), Felix Schönbrodt (0000-0002-8282-3910, Ludwig-Maximilians-Universität München), Felix D Schönbrodt (Ludwig-Maximilians-Universität München), Will M Gervais (0000-0001-7790-1665, University of Kentucky), Joseph Hilgard (0000-0002-7278-4698, Illinois State University) |
| Year | 2019 |
| Volume | 2 |
| Issue | 2 |
| Pages | 115-144 |
| Publication date | 2019-06-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Advances in Methods and Practices in Psychological Science (JOURNAL) |
| Journal identifiers | ISSN: 2515-2459 • E-ISSN: 2515-2467 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/2515245919847196 |
| OpenAlex | W2781957993 |
| Language | EN |
| Citations received | 103 |
| References cited | 74 |
Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to correct for such overestimation. However, it is not clear which methods work best for data typically seen in psychology. Here, we present a comprehensive simulation study in which we examined how some of the most promising meta-analytic methods perform on data that might realistically be produced by research in psychology. We simulated several levels of questionable research practices, publication bias, and heterogeneity, and used study sample sizes empirically derived from the literature. Our results clearly indicated that no single meta-analytic method consistently outperformed all the others. Therefore, we recommend that meta-analysts in psychology focus on sensitivity analyses—that is, report on a variety of methods, consider the conditions under which these methods fail (as indicated by simulation studies such as ours), and then report how conclusions might change depending on which conditions are most plausible. Moreover, given the dependence of meta-analytic methods on untestable assumptions, we strongly recommend that researchers in psychology continue their efforts to improve the primary literature and conduct large-scale, preregistered replications. We provide detailed results and simulation code at https://osf.io/rf3ys and interactive figures at http://www.shinyapps.org/apps/metaExplorer/ .
Data science · Econometrics · Focus (optics) · Management science · Meta-analysis · Psychological research · Publication bias · Sample size determination · Statistics · Variety (cybernetics) · Applied Psychology · Artificial Intelligence · Behavioral Health and Interventions · Computer Science · Mathematics · Mental Health Research Topics · Meta-analysis and systematic reviews · Psychology · Social Psychology
Burnout en docentes. Meta-análisis a partir del Maslach Burnout Inventory
Relationship between mental health help-seeking intention and prospective help-seeking behavior
Toward a comprehensive and potentially cross-cultural model of why people engage in collective action
Is bilingualism related to a cognitive advantage in children? A systematic review and meta-analysis
Heterogeneity in direct replications in psychology and its association with effect size
Testing (quizzing) boosts classroom learning
Is bilingualism associated with enhanced executive functioning in adults? A meta-analytic review
Promoting Transparent Reporting of Conflicts of Interests and Statistical Analyses at The Journal of Sex Research
Is a procedural learning deficit a causal risk factor for developmental language disorder or dyslexia? A meta-analytic review
Shared education as a contact-based intervention to improve intergroup relations among adolescents in postconflict Northern Ireland
The association between weight stigma and mental health
Machine learning uncovers the most robust self-report predictors of relationship quality across 43 longitudinal couples studies
Loneliness and the Big Five Personality Traits
Replicability, Robustness, and Reproducibility in Psychological Science
The Gamification of Learning
Long-term Persuasive Effects in Narrative Communication Research
Meta-Analysis of Present-Bias Estimation using Convex Time Budgets
Meta-Analysis in Organizational Research
Integrating exploration and prediction in computational psychotherapy science
Seductive Details, Cognitive Load, and Learning Outcomes
The current phase of social sciences research
Intuitions About the Reference of Proper Names
The Cooperation Databank
Sorting the File Drawer
Commentary on Elkjær et al.’s (2020) Meta-Analysis on Expansive Versus Contractive Nonverbal Displays
Rural electrification, the credibility revolution, and the limits of evidence-based policy
Is probabilistic cuing of visual search an inflexible attentional habit? A meta-analytic review
Correcting for publication bias in a meta-analysis with the p-uniform* method
Does bilingualism come with linguistic costs? A meta-analytic review of the bilingual lexical deficit
Can I trust this paper
Category learning in autistic individuals
The relationship between mind wandering and reading comprehension
Effect of episodic specificity induction on divergent thinking
The effectiveness of digital interventions for enhancing empathy in adults
Central questions about meta-analyses in psychological research
How and when transformational leadership sparks innovative workplace behaviors
Creativity in the West and the East
Investigating the ease-of-retrieval effect in an eyewitness context
Illness and treatment beliefs and health outcomes in chronic pain
The impact of open educational resources on student achievement
The effects of language learning strategy instruction on college students’ English achievement and learner autonomy in mainland China
Are risk assessment tools more accurate than unstructured judgments in predicting violent, any, and sexual offending? A meta‐analysis of direct comparison studies
A Meta-Analysis of the Effectiveness of EMDR and TF-CBT in Reducing Trauma Symptoms and Externalizing Behavior Problems in Adolescents
P -Curving as a Safeguard Against P -Hacking in Sla Research
Impact of past behaviour normality
Impact of Integrating Computational Thinking in STEM Education on Students' Cognitive and Non‐Cognitive Skills
Children’s domain-specific self-evaluations and global self-worth
Evaluating the Evidence for Enclothed Cognition
Narcissism and Wellbeing
Spontaneous Trait Inferences From Behavior
Vicarious Dissonance
Observation Moderates the Moral Licensing Effect
The Behavioral Impact of Moral Appeals
Does Mindfulness Improve Intergroup Bias, Internalized Bias, and Anti-Bias Outcomes
Revisiting Communication Infrastructure Theory
Physical Appearance Anxiety and Eating Disorders Symptomatology
Simulation-Based Learning in Higher Education
“Yes, I Can!” A Systematic Review and Meta-Analysis of Intervention Studies Promoting Teacher Self-Efficacy
Mind the Gap Between Comprehension and Metacomprehension
The Impact of Test Preparation on Performance of Large-Scale Educational Tests
The Montessori Method Under Scrutiny
How Strongly Is Teachers’ Burnout Related to Teacher Absenteeism, Teacher–Student Interactions, and Student Motivation and Achievement
Effectiveness of conceptual change strategies in science education
Research Review
The future is present in the past
The effectiveness of implementation intentions in children
Critically evaluating and advancing research and statistical methods in vocational behavior research
A meta-analysis on air traffic controllers selection
Meta-analysis in vocational behavior
Saved by the biodata
How credible are published significant results
A meta-analysis of teacher language practices in early childhood classrooms
Mortality salience
Insecure attachment as a predictor of death anxiety
Does FDI promote entrepreneurial activities? A meta-analysis
Fairness Hacking
Early Psychological Interventions for Somatic Symptom Disorder and Functional Somatic Syndromes
The Effects of Political Incivility on Political Trust and Political Participation
Emotion Regulation and Aggression
The effect of bilingualism and multicultural experience on social-cognitive processing
Emotion regulation and mental health across cultures
An umbrella review of randomized control trials on the effects of physical exercise on cognition
Perceived economic hardship and adjustment outcomes of children and adolescents
Longitudinal tests of the theory of planned behaviour
Social cognition correlates of self-management behaviors in patients with familial hypercholesterolemia (FH)
When is narcissism associated with low empathy? A meta-analytic review
The discriminant validity of honesty-humility
Publication bias
Owning leads to valuing
The help that hinders? A meta‐analysis of reactions to affirmative action
Evidence against subliminal anchoring
Math approach training changes implicit identification with math
What is Humility Good For? A Meta-Analysis of Humility, Health, and Well-Being
Assessment of Evidential Value Requires More than a Single Data Point
Reply to
How Political Efficacy Relates to Online and Offline Political Participation
Nudging Social Media toward Accuracy
The Past, Present, and Future of Ego Depletion
Reducing bias, increasing transparency and calibrating confidence with preregistration
A multi-lab test of the facial feedback hypothesis by the Many Smiles Collaboration
Introduction to Meta‐Analysis
Introduction to Meta‐Analysis
The Handbook of Research Synthesis and Meta-Analysis
The Rules of the Game Called Psychological Science
Methods to estimate the between‐study variance and its uncertainty in meta‐analysis
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
Continuously Cumulating Meta-Analysis and Replicability
Publication Bias in Research Synthesis
Publication Bias in Psychology
A series of meta-analytic tests of the depletion effect
A Vast Graveyard of Undead Theories
Meta-regression approximations to reduce publication selection bias
P -Curve and Effect Size
Adjusting for Publication Bias in Meta-Analysis
Publication and related bias in meta-analysis
A Multilab Preregistered Replication of the Ego-Depletion Effect
Meta-analyses are no substitute for registered replications
Correlational effect size benchmarks.
Negative results are disappearing from most disciplines and countries
Trim and Fill
Bias in meta-analysis detected by a simple, graphical test
False-Positive Psychology
Conducting Meta-Analyses in R with the metafor Package
Limitations of PET-Peese and Other Meta-Analysis Methods
Underreporting in Psychology Experiments
One Hundred Years of Social Psychology Quantitatively Described
Many Labs 2
The Power of Bias in Economics Research
Conceptualizing and evaluating the replication of research results
Self-control as a limited resource
Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills
What meta-analyses reveal about the replicability of psychological research
Overstated evidence for short-term effects of violent games on affect and behavior
Consequences of prejudice against the null hypothesis
Ego depletion and the strength model of self-control
| Unique citing works | 103 |
|---|---|
| Citations per year | 12,88 |
| Citation span | 2018 - 2026 (9) |
| Citation velocity | current |
| Highly cited | Yes |
| Citation types | Neutral: 93 |