Why psychologists must change the way they analyze their data
The case of psi: Comment on Bem (2011)
Bibliographic Data
| ID | 6198570 |
|---|---|
| Authors | Eric-Jan Wagenmakers (0000-0003-1596-1034, University of Amsterdam), Ruud Wetzels (0000-0002-4232-8288), Denny Borsboom (0000-0001-9720-4162), Han L J Van Der Maas (0000-0001-8278-319X) |
| Year | 2011 |
| Volume | 100 |
| Issue | 3 |
| Pages | 426-432 |
| Publication date | 2011-03-01 |
| Peer Reviewed | Yes |
| Open Access | No |
| Type | ARTICLE |
| Venue | Journal of Personality and Social Psychology (JOURNAL) |
| Journal identifiers | ISSN: 0022-3514 • E-ISSN: 1939-1315 |
| Publisher | American Psychological Association (APA) (PUBLISHER) |
| DOI | 10.1037/a0022790 |
| PMID | 21280965 |
| OpenAlex | W2169244873 |
| Language | EN |
| Citations received | 139 |
| References cited | 8 |
Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data
Affect (linguistics) · Bayesian probability · Cognitive psychology · Econometrics · Epistemology · Null hypothesis · Skepticism · Statistical hypothesis testing · Statistics · Test (biology) · Decision-Making and Behavioral Economics · Mathematics · Paranormal Experiences and Beliefs · Philosophy · Psychology · Psychology of Moral and Emotional Judgment · Social Psychology
Who punishes promiscuous women? Both women and men are prejudiced towards sexually-accessible women, but only women inflict costly punishment
Registered Reports as a Method to Increase Credibility of Science – Experimental Study Among Psychological Students
Psychological science’s replicability crisis and what it means for science in the courtroom
Addiction is a brain disease, and it doesn’t matter
Exposing People to Others Not Following Recommendations Reduces Unrealistic Optimism. An Experiment During a Pandemic
On selecting evidence to test hypotheses
Perceived crisis and reforms
Treating stimuli as a random factor in social psychology
Must psychologists change the way they analyze their data
Correcting the past
Reactive and proactive interference control in adults with autism spectrum disorder across the lifespan
Oxford Studies in Philosophy of Religion
Bayesian Cognitive Modeling
Statistical Inference as Severe Testing
The ironic effect of significant results on the credibility of multiple-study articles.
The fallacy of placing confidence in confidence intervals
The Rules of the Game Called Psychological Science
Bayesian inference for psychology. Part I
On the Interpretation and Use of Mediation
Statistical Evidence in Experimental Psychology
Psychology's Renaissance
Phubbing behavior in conversations and its relation to perceived conversation intimacy and distraction
Unconscious influences on decision making
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
A Vast Graveyard of Undead Theories
Replicability, Robustness, and Reproducibility in Psychological Science
An Agenda for Purely Confirmatory Research
Recommendations for Increasing Replicability in Psychology
Replications in Psychology Research
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies
Is the Replicability Crisis Overblown? Three Arguments Examined
The natural selection of bad science
Increasing Transparency Through a Multiverse Analysis
Scientific Utopia
A Gentle Introduction to Bayesian Analysis
A Large Scale Test of the Effect of Social Class on Prosocial Behavior
False-Positive Psychology
Aging and the Social Ecology of Everyday Interpersonal Perception
Empirical Investigation of the Verbal Cues Involved in Delivering Experiential Metaphors
Applying behavioural science to increase uptake of the NHS Health Check
Acceptance and commitment therapy in parents of children with cancer at psychosocial risk
A multiple-baseline evaluation of a brief acceptance and commitment therapy protocol focused on repetitive negative thinking for moderate emotional disorders
The effect of transcranial direct current stimulation on frontal alpha asymmetry and visuospatial attention in food-reward contexts
Neuroticism Drives Associations Between Repetitive Behaviors and Depression in Autistic Adults
After a pair of self-control-intensive tasks, sucrose swishing improves subsequent working memory performance
Advancing the Cognitive Science of Religion through Replication and Open Science
A Primer for Conducting Experiments in Human–Robot Interaction
A Journey Through Philosophy and Medicine
Causal effects of social media use on self-esteem, mindfulness, sleep and emotional well-being
Significance Tests
Theory Construction Methodology
Shocking advantage! Improving digital game performance using non-invasive brain stimulation
The disintegration of event files over time
The role of episodic memory sampling in evaluation
Exploring mechanisms underlying intertemporal choices
Open science practices in the false memory literature
Intact strategic retrieval processes in older adults
The animacy (bias) effect in recognition
Beware influential findings that have not been replicated
Are graphemic effects real in skilled visual word recognition
How does semantic knowledge impact working memory maintenance? Computational and behavioral investigations
Introduction to the Special Series
Music in Mood Regulation Brief Scale (B-MMR)
Psychological science from 'publish or perish' to 'trust but verify'
Meaningful boundaries create boundary conditions for control
Registered report “Categorical perception of facial expressions of anger and disgust across cultures”
The relationships between perceived teacher autonomy support, academic self-efficacy and learning engagement among primary school students
There is Nothing Magical about Bayesian Statistics
Bias analysis in forensic and non-forensic psychiatric assessments
Gray (Literature) Matters
Acceptance and Commitment Therapy Focused on Repetitive Negative Thinking for Child Depression
Enhancing literacy acquisition through spelling exercises
Forest before trees? It depends on not only what you see, but also what you hear
Attentional fluctuations and the temporal organization of memory
Validity of attention self-reports in younger and older adults
Implicit causal inference in audiovisual spatial representations
How many seconds was that? Teaching children about time does not refine their ability to track durations
I want to know why this feels so familiar
Patterns of bilingual language use and response inhibition
Who's the winner? Children's math learning in competitive and collaborative scenarios
Getting Serious About the Assessment and Promotion of Replicable Sexual Science
Facial mimicry is independent of stimulus format
Do Easterners and Westerners Differ in Visual Cognition? A Preregistered Examination of Three Visual Cognition Tasks
On the category adjustment model
Ethical Consistency and Experience
How to improve people's interpretation of probabilities of precipitation
Promoting pro-environmental attitudes and reported behaviors of Malaysian pre-service teachers using green chemistry experiments
Metaphysical Chauvinism and Perceptions of Deviance in Religion, Atheism, and Alien Realms
The Use and Misuse of Classical Statistics
Priming Risky Choice
An Appeal for a Methodological Fusion of Conversation Analysis and Experimental Psychology
Independent operation of implicit working memory under cognitive load
Afterthoughts on precognition
Observation-Oriented Modeling
Performing Contrast Analysis in Factorial Designs
Do the disadvantaged legitimize the social system? A large-scale test of the status–legitimacy hypothesis
Failures to change stimulus evaluations by means of subliminal approach and avoidance training
The state of social and personality science
Innocent suspects’ willingness to disclose critical information
Is Social Work Research in Crisis
| Unique citing works | 139 |
|---|---|
| Citations per year | 9,27 |
| Citation span | 2011 - 2026 (16) |
| Citation velocity | current |
| Highly cited | Yes |
| Citation types | Neutral: 128 |