Testing Accuracy and Agreement among Multiple Versions of Automated Bat Call Classification Software
- 20 December 2021
- journal article
- research article
- Published by Wiley in Wildlife Society Bulletin
- Vol. 45 (4) , 690-705
Abstract
Passive acoustic monitoring is a common method of studying bats that involves recording echolocation calls of bats in their natural environment. Call sequences are then identified to species using automated acoustic analysis software. One limitation of acoustic software programs, particularly for long‐term monitoring efforts, is that newer versions may provide results that are not directly comparable with older versions. However, there is little available information regarding how much or in what ways the versions differ, or which versions are most accurate. We evaluated 2 software programs used for automated bat call identification by testing a common set of echolocation call files across multiple versions of each program. We quantified the level of agreement on identification results and compared accuracy rates among the versions of each program. Level of agreement varied by species, recording location, and the software versions being compared. Overall percent agreement ranged from 28–82%. Newer versions were more conservative, in that they assigned fewer species‐level identifications. However, newer versions were not substantially more accurate than older versions. Our conclusions suggest that bat researchers should be attentive to what software versions and settings are used as they plan and perform data analyses. Software developers could assist software users by providing more detailed information about their testing procedures and results, and what changes are associated with new versions. © 2021 The Wildlife Society.Keywords
Funding Information
- North Dakota State University
- National Park Service
This publication has 40 references indexed in Scilit:
- A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samplesBMC Medical Research Methodology, 2013
- Current state of understanding of ultrasonic detectors for the study of bat ecologyMammal Research, 2013
- Autonomous Recording of Great Gray Owls in the Sierra NevadaNorthwestern Naturalist, 2012
- Sampling period, size and duration influence measures of bat species richness from acoustic surveysMethods in Ecology and Evolution, 2012
- Patterns of Acoustical Activity of Bats Prior to and Following White-Nose Syndrome OccurrenceJournal of Fish and Wildlife Management, 2011
- Human vs. machine: identification of bat species from their echolocation calls by humans and by artificial neural networksCanadian Journal of Zoology, 2008
- Assessing Bat Detectability and Occupancy with Multiple Automated Echolocation DetectorsJournal of Mammalogy, 2008
- Bats are Not Birds--a Cautionary Note on Using Echolocation Calls to Identify Bats: a CommentJournal of Mammalogy, 1999
- Bias, prevalence and kappaJournal of Clinical Epidemiology, 1993
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960