An Autocorrelation-based LSTM-Autoencoder for Anomaly Detection on Time-Series Data
Authors:
- 10 December 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5068-5077
Abstract
Data quality significantly impacts the results of data analytics. Researchers have proposed machine learning based anomaly detection techniques to identify incorrect data. Existing approaches fail to (1) identify the underlying domain constraints violated by the anomalous data, and (2) generate explanations of these violations in a form comprehensible to domain experts. We propose IDEAL, which is an LSTM-Autoencoder based approach that detects anomalies in multivariate time-series data, generates domain constraints, and reports subsequences that violate the constraints as anomalies. We propose an automated autocorrelation-based windowing approach to adjust the network input size, thereby improving the correctness and performance of constraint discovery over manual and brute-force approaches. The anomalies are visualized in a manner comprehensible to domain experts in the form of decision trees extracted from a random forest classifier. Domain experts can then provide feedback to retrain the learning model and improve the accuracy of the process. We evaluate the effectiveness of IDEAL using datasets from Yahoo servers, NASA Shuttle, and Colorado State University Energy Institute. We demonstrate that IDEAL can detect previously known anomalies from these datasets. Using mutation analysis, we show that IDEAL can detect different types of injected faults. We also demonstrate that the accuracy improves after incorporating domain expert feedback.Keywords
This publication has 29 references indexed in Scilit:
- Anomaly Detection with Robust Deep Autoencoders2017
- A framework for sensitivity analysis of decision treesCentral European Journal of Operations Research, 2017
- Application of one-class support vector machine to quickly identify multivariate anomalies from geochemical exploration dataGeochemistry: Exploration, Environment, Analysis, 2017
- An approach for classification using simple CART algorithm in WEKA2017
- Robust Online Time Series Prediction with Recurrent Neural Networks2016
- Deep Learning over Multi-field Categorical Data2016
- Large-Scale Unusual Time Series Detection2015
- Generic and Scalable Framework for Automated Time-series Anomaly Detection2015
- An Interactive Approach to Outlier DetectionLecture Notes in Computer Science, 2010
- LOF2000