Reducing Overconfidence in the Interval Judgments of Experts

Dados Bibliográficos

ID19609166
AutoresAndrew Speirs‐Bridge (La Trobe University), Andrew Speirs-Bridge, Fiona Fidler (0000-0002-2700-2562, La Trobe University), Marissa F Mcbride (0000-0002-7498-292X), Marissa McBride, Louisa Flander (0000-0003-0824-9364, The University of Melbourne), Geoff Cummings (0000-0003-1716-5277, La Trobe University), GEOFF CUMMING, Mark A Burgman (0000-0003-0435-4829, autor correspondente), Mark Burgman
Ano2009
Volume30
Fascículo3
Páginas512-523
Data de publicação2009-12-17
Peer ReviewedSim
Open AccessSim
TipoARTICLE
PeriódicoRisk Analysis (JOURNAL)
Identificadores do periódicoISSN: 0272-4332 • E-ISSN: 1539-6924
EditoraWiley (PUBLISHER • GB)
DOI10.1111/j.1539-6924.2009.01337.x
PMID20030766
OpenAlexW1966924126
IdiomaEN
Citações recebidas17
Referências citadas19

Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3-point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4-step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta-analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)-a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4-step procedure is more likely to reduce overconfidence than the 3-point procedure (Cohen's d = 0.61, [0.04, 1.18])

Actuarial science · Confidence interval · Econometrics · Economics · Expert elicitation · Overconfidence effect · Point estimation · Statistics · Forecasting Techniques and Applications · Mathematics · Meta-analysis and systematic reviews · Psychology · Risk Perception and Management · Social Psychology