Expert agreement in prior elicitation and its effects on Bayesian inference
Bibliographic Data
| ID | 21640697 |
|---|---|
| Authors | Angelika Marlene Stefan (0000-0003-3382-4746, Amsterdam University of Applied Sciences, corresponding author), Dimitris Katsimpokis (0000-0002-4073-0206, University of Basel, corresponding author), Quentin F Gronau (0000-0001-5510-6943, University of Amsterdam), Eric-Jan Wagenmakers (0000-0003-1596-1034, University of Amsterdam) |
| Year | 2022 |
| Volume | 29 |
| Issue | 5 |
| Pages | 1776-1794 |
| Publication date | 2022-10-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Psychonomic Bulletin & Review (JOURNAL) |
| Journal identifiers | ISSN: 1069-9384 • E-ISSN: 1531-5320 |
| Publisher | Springer Science and Business Media LLC (PUBLISHER) |
| DOI | 10.3758/s13423-022-02074-4 |
| PMID | 35378671 |
| OpenAlex | W4226133478 |
| Language | EN |
| Citations received | 1 |
| References cited | 64 |
Bayesian inference requires the specification of prior distributions that quantify the pre-data uncertainty about parameter values. One way to specify prior distributions is through prior elicitation, an interview method guiding field experts through the process of expressing their knowledge in the form of a probability distribution. However, prior distributions elicited from experts can be subject to idiosyncrasies of experts and elicitation procedures, raising the spectre of subjectivity and prejudice. Here, we investigate the effect of interpersonal variation in elicited prior distributions on the Bayes factor hypothesis test. We elicited prior distributions from six academic experts with a background in different fields of psychology and applied the elicited prior distributions as well as commonly used default priors in a re-analysis of 1710 studies in psychology. The degree to which the Bayes factors vary as a function of the different prior distributions is quantified by three measures of concordance of evidence: We assess whether the prior distributions change the Bayes factor direction, whether they cause a switch in the category of evidence strength, and how much influence they have on the value of the Bayes factor. Our results show that although the Bayes factor is sensitive to changes in the prior distribution, these changes do not necessarily affect the qualitative conclusions of a hypothesis test. We hope that these results help researchers gauge the influence of interpersonal variation in elicited prior distributions in future psychological studies. Additionally, our sensitivity analyses can be used as a template for Bayesian robustness analyses that involve prior elicitation from multiple experts
Agreement · Bayesian inference · Bayesian probability · Bayesian statistics · Cognitive psychology · Expert elicitation · Inference · Linguistics · Statistics · Computer Science · Decision-Making and Behavioral Economics · Meta-analysis and systematic reviews · Psychology · Statistical Methods and Bayesian Inference · Artificial Intelligence
Uncertain Judgements
Stan
Bayesian inference for psychology. Part I
Statistical Evidence in Experimental Psychology
Practical Bayesian model evaluation using leave-one-out cross-validation and Waic
Extensional versus intuitive reasoning
Improving transparency and replication in Bayesian statistics
Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison
Bayes factor design analysis
Increasing Transparency Through a Multiverse Analysis
Registered Reports
Correlational effect size benchmarks.
Bayes Factors
Statistical Power Analysis
The Jasp guidelines for conducting and reporting a Bayesian analysis
Novel Evidence and Severe Tests
Mindless statistics
| Unique citing works | 1 |
|---|---|
| Citations per year | 1 |
| Citation span | 2025 - 2025 (1) |
| Citation velocity | recent |
| Highly cited | No |
| Citation types | Neutral: 1 |