Online System Problem Detection by Mining Patterns of Console Logs
Authors:
- Wei XuWXWei Xu
- Ling HuangLHLing Huang
- Armando FoxAFArmando Fox
- David PattersonDPDavid Patterson
- Michael JordanMJMichael Jordan
Top Cited Papers
- 1 December 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15504786,p. 588-597
Abstract
We describe a novel application of using data mining and statistical learning methods to automatically monitor and detect abnormal execution traces from console logs in an online setting. Different from existing solutions, we use a two stage detection system. The first stage uses frequent pattern mining and distribution estimation techniques to capture the dominant patterns (both frequent sequences and time duration). The second stage use principal component analysis based anomaly detection technique to identify actual problems. Using real system data from a 203-node Hadoop cluster, we show that we can not only achieve highly accurate and fast problem detection, but also help operators better understand execution patterns in their system.Keywords
This publication has 17 references indexed in Scilit:
- Power-Law Distributions in Empirical DataSIAM Review, 2009
- Power laws in softwareACM Transactions on Software Engineering and Methodology, 2008
- Frequent pattern mining for kernel trace data2008
- IMDS2007
- Profiling the End Host2007
- Frequent pattern mining: current status and future directionsData Mining and Knowledge Discovery, 2007
- Profiling internet backbone trafficACM SIGCOMM Computer Communication Review, 2005
- Dynamic syslog mining for network failure monitoring2005
- On power-law relationships of the Internet topologyACM SIGCOMM Computer Communication Review, 1999
- Control Procedures for Residuals Associated With Principal Component AnalysisTechnometrics, 1979