Reducing Overconfidence in the Interval Judgments of Experts

Bibliographic Data

ID19609166
AuthorsAndrew Speirs‐Bridge (La Trobe University), Andrew Speirs-Bridge, Fiona Fidler (0000-0002-2700-2562, La Trobe University), Marissa F Mcbride (0000-0002-7498-292X), Marissa McBride, Louisa Flander (0000-0003-0824-9364, The University of Melbourne), Geoff Cummings (0000-0003-1716-5277, La Trobe University), GEOFF CUMMING, Mark A Burgman (0000-0003-0435-4829, corresponding author), Mark Burgman
Year2009
Volume30
Issue3
Pages512-523
Publication date2009-12-17
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueRisk Analysis (JOURNAL)
Journal identifiersISSN: 0272-4332 • E-ISSN: 1539-6924
PublisherWiley (PUBLISHER • GB)
DOI10.1111/j.1539-6924.2009.01337.x
PMID20030766
OpenAlexW1966924126
LanguageEN
Citations received17
References cited19

Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3-point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4-step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta-analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)-a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4-step procedure is more likely to reduce overconfidence than the 3-point procedure (Cohen's d = 0.61, [0.04, 1.18])

Actuarial science · Confidence interval · Econometrics · Economics · Expert elicitation · Overconfidence effect · Point estimation · Statistics · Forecasting Techniques and Applications · Mathematics · Meta-analysis and systematic reviews · Psychology · Risk Perception and Management · Social Psychology