Extracting the collective wisdom in probabilistic judgments
Bibliographic Data
| ID | 21405017 |
|---|---|
| Authors | Cem Peker (0000-0001-9036-1915, Erasmus University Rotterdam, corresponding author) |
| Year | 2023 |
| Volume | 94 |
| Issue | 3 |
| Pages | 467-501 |
| Publication date | 2023-04-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Theory and Decision (JOURNAL) |
| Journal identifiers | ISSN: 0040-5833 • E-ISSN: 1573-7187 |
| Publisher | Springer Science and Business Media LLC (PUBLISHER) |
| DOI | 10.1007/s11238-022-09899-4 |
| OpenAlex | W4292452059 |
| Language | EN |
| References cited | 28 |
How should we combine disagreeing expert judgments on the likelihood of an event? A common solution is simple averaging, which allows independent individual errors to cancel out. However, judgments can be correlated due to an overlap in their information, resulting in a miscalibration in the simple average. Optimal weights for weighted averaging are typically unknown and require past data to estimate reliably. This paper proposes an algorithm to aggregate probabilistic judgments under shared information. Experts are asked to report a prediction and a meta-prediction. The latter is an estimate of the average of other individuals’ predictions. In a Bayesian setup, I show that if average prediction is a consistent estimator, the percentage of predictions and meta-predictions that exceed the average prediction should be the same. An “overshoot surprise” occurs when the two measures differ. The Surprising Overshoot algorithm uses the information revealed in an overshoot surprise to correct for miscalibration in the average prediction. Experimental evidence suggests that the algorithm performs well in moderate to large samples and in aggregation problems where individuals disagree in their predictions
Aggregate (composite) · Algorithm · Bayesian probability · Econometrics · Estimator · Event (particle physics) · Overshoot (microwave communication) · Probabilistic logic · Scoring rule · Simple (philosophy) · Statistics · Surprise · Artificial Intelligence · Bayesian Modeling and Causal Inference · Computer Science · Decision-Making and Behavioral Economics · Forecasting Techniques and Applications · Mathematics · Psychology · Social Psychology
| Highly cited | No |
|---|