Evaluating presence–absence models in ecology: the need to account for prevalence
Authors:
Top Cited Papers
- 1 October 2001
- journal article
- research article
- Published by Wiley in Journal of Applied Ecology
- Vol. 38 (5) , 921-931
Abstract
Summary: Models for predicting the distribution of organisms from environmental data are widespread in ecology and conservation biology. Their performance is invariably evaluated from the percentage success at predicting occurrence at test locations. Using logistic regression with real data from 34 families of aquatic invertebrates in 180 Himalayan streams, we illustrate how this widespread measure of predictive accuracy is affected systematically by the prevalence (i.e. the frequency of occurrence) of the target organism. Many evaluations of presence–absence models by ecologists are inherently misleading. With the same invertebrate models, we examined alternative performance measures used in remote sensing and medical diagnostics. We particularly explored receiver‐operating characteristic (ROC) plots, from which were derived (i) the area under each curve (AUC), considered an effective indicator of model performance independent of the threshold probability at which the presence of the target organism is accepted, and (ii) optimized probability thresholds that maximize the percentage of true absences and presences that are correctly identified. We also evaluated Cohen's kappa, a measure of the proportion of all possible cases of presence or absence that are predicted correctly after accounting for chance effects. AUC measures from ROC plots were independent of prevalence, but highly significantly correlated with the much more easily computed kappa. Moreover, when applied in predictive mode to test data, models with thresholds optimized by ROC erroneously overestimated true occurrence among scarcer organisms, often those of greatest conservation interest. We advocate caution in using ROC methods to optimize thresholds required for real prediction. Our strongest recommendation is that ecologists reduce their reliance on prediction success as a performance measure in presence–absence modelling. Cohen's kappa provides a simple, effective, standardized and appropriate statistic for evaluating or comparing presence–absence models, even those based on different statistical algorithms. None of the performance measures we examined tests the statistical significance of predictive accuracy, and we identify this as a priority area for research and development.Keywords
This publication has 58 references indexed in Scilit:
- PREDICTING THE LIKELIHOOD OF EURASIAN WATERMILFOIL PRESENCE IN LAKES, A MACROPHYTE MONITORING TOOLEcological Applications, 2000
- Large‐scale processes in ecology and hydrologyJournal of Applied Ecology, 2000
- Predicting the spatial distribution of non‐indigenous riparian weeds: issues of spatial scale and extentJournal of Applied Ecology, 2000
- The uncertainty in simulations by a Global Biome Model (BIOME3) to alternative parameter valuesGlobal Change Biology, 2000
- Statistical Power of Presence‐Absence Data to Detect Population DeclinesConservation Biology, 1999
- The impact of American mink Mustela vison on water birds in the upper ThamesJournal of Applied Ecology, 1999
- A Critique for MacroecologyOikos, 1999
- PREDICTION OF RARE-PLANT OCCURRENCE: A SOUTHERN APPALACHIAN EXAMPLEEcological Applications, 1998
- The impact of agricultural land use on stream chemistry in the Middle Hills of the Himalayas, NepalJournal of Hydrology, 1996
- Current approaches to modelling the environmental niche of eucalypts: implication for management of forest biodiversityForest Ecology and Management, 1996