ADELE: Anomaly Detection from Event Log Empiricism
Authors:
- 1 April 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2114-2122
Abstract
A large population of users gets affected by sudden slowdown or shutdown of an enterprise application. System administrators and analysts spend considerable amount of time dealing with functional and performance bugs. These problems are particularly hard to detect and diagnose in most computer systems, since there is a huge amount of system generated supportability data (counters, logs etc.) that need to be analyzed. Most often, there isn't a very clear or obvious root cause. Timely identification of significant change in application behavior is very important to prevent negative impact on the service. In this paper, we present ADELE, an empirical, data-driven methodology for early detection of anomalies in data storage systems. The key feature of our solution is diligent selection of features from system logs and development of effective machine learning techniques for anomaly prediction. ADELE learns from system's own history to establish the baseline of normal behavior and gives accurate indications of the time period when something is amiss for a system. Validation on more than 4800 actual support cases shows ~ 83% true positive rate and ~ 12% false positive rate in identifying periods when the machine is not performing normally. We also establish the existence of problem “signatures” which help map customer problems to already seen issues in the field. ADELE's capability to predict early paves way for online failure prediction for customer systems.Keywords
This publication has 9 references indexed in Scilit:
- Learning a Hierarchical Monitoring System for Detecting and Diagnosing Service Issues2015
- Generic and Scalable Framework for Automated Time-series Anomaly Detection2015
- Real-time failure prediction in online services2015
- Anode: Empirical detection of performance problems in storage systems using time-series analysis of periodic measurements2014
- Failure Prediction in IBM BlueGene/L Event Logs2007
- A Meta-Learning Failure Predictor for Blue Gene/L Systems2007
- Filtering Failure Logs for a BlueGene/L Prototype2005
- A Survey of Outlier Detection MethodologiesArtificial Intelligence Review, 2004
- Algorithm AS 136: A K-Means Clustering AlgorithmJournal of the Royal Statistical Society Series C: Applied Statistics, 1979