Assessing the accuracy of species distribution models

Prevalence, kappa and the true skill statistic (TSS)

Bibliographic Data

ID23327342
AuthorsOMRI ALLOUCHE (0000-0002-4595-8242, Hebrew University of Jerusalem, corresponding author), ASAF TSOAR (Hebrew University of Jerusalem), RONEN KADMON (0000-0001-7637-4058, Hebrew University of Jerusalem)
Year2006
Volume43
Issue6
Pages1223-1232
Publication date2006-12-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueJournal of Applied Ecology (JOURNAL)
Journal identifiersISSN: 0021-8901 • E-ISSN: 1365-2664
PublisherWiley (PUBLISHER • GB)
DOI10.1111/j.1365-2664.2006.01214.x
OpenAlexW1541774929
LanguageEN
Citations received90
References cited67

In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence–absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion. In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel. The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence. When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence. Synthesis and applications . Our results provide theoretical and empirical evidence that kappa, one of the most widely used measures of model performance in ecology, has serious limitations that make it unsuitable for such applications. The alternative we suggest, TSS, compensates for the shortcomings of kappa while keeping all of its advantages. We therefore recommend the TSS as a simple and intuitive measure for the performance of species distribution models when predictions are expressed as presence–absence maps.

Cohen's kappa · Contrast (vision) · Data mining · Econometrics · Kappa · Measure (data warehouse) · Statistic · Statistics · Artificial Intelligence · Computer Science · Ecology and Vegetation Dynamics Studies · Mathematics · Species Distribution and Climate Change · Wildlife Ecology and Conservation

Tools

Open DOISci-HubOpen Access