Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series
Open Access
- 18 November 2023
- journal article
- research article
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 38 (3) , 1027-1068
Abstract
The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate method for a specific domain. The evaluation of these methods is facilitated by the use of metrics, which vary widely in their properties. Despite the existence of new evaluation metrics, there is limited agreement on which metrics are best suited for specific scenarios and domains, and the most commonly used metrics have faced criticism in the literature. This paper provides a comprehensive overview of the metrics used for the evaluation of time series anomaly detection methods, and also defines a taxonomy of these based on how they are calculated. By defining a set of properties for evaluation metrics and a set of specific case studies and experiments, twenty metrics are analyzed and discussed in detail, highlighting the unique suitability of each for specific tasks. Through extensive experimentation and analysis, this paper argues that the choice of evaluation metric must be made with care, taking into account the specific requirements of the task at hand.Keywords
All Related Versions
Funding Information
- Norges Forskningsråd (312062)
- SINTEF
This publication has 85 references indexed in Scilit:
- ADE: An ensemble approach for early Anomaly Detection2017
- Evaluating Real-Time Anomaly Detection Algorithms -- The Numenta Anomaly Benchmark2015
- The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced DatasetsPLOS ONE, 2015
- Robust and Accurate Anomaly Detection in ECG Artifacts Using Time Series Motif DiscoveryComputational and Mathematical Methods in Medicine, 2015
- Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)Briefings in Bioinformatics, 2011
- AUC: a misleading measure of the performance of predictive distribution modelsGlobal Ecology and Biogeography, 2007
- ECG Anomaly Detection via Time Series Analysis2007
- Finding Unusual Medical Time-Series Subsequences: Algorithms and ApplicationsIEEE Transactions on Information Technology in Biomedicine, 2006
- The relationship between Precision-Recall and ROC curves2006
- A Proposed Design and Analysis for Comparing Digital and Analog MammographyJournal of the American Statistical Association, 2001