Proactive failure detection learning generation patterns of large-scale network logs
Authors:
- 1 November 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8-14
Abstract
With the growth of services in IP networks, network operators are required to perform proactive operation that quickly detects the signs of critical failures and prevents future problems. Network log data, including router syslog, are rich sources for such operations. However, it has become impossible to find genuinely important logs that lead to serious problems due to the large volume and complexity of log data. We propose a log analysis system for proactive detection of failures. Our key observation is that the abnormality of logs depends on not just the keywords in the messages (e.g. ERROR, FAIL), but generation patterns such as burstiness. Our system consists of three functions: (i) extracting log templates automatically and quickly from a massive amount of unstructured log data; (ii) constructing log feature vectors to characterize the generation patterns of logs; and (iii) using a supervised machine learning approach to associate failures with the log data that appeared before them. We validated our system using real log data collected from a large network and determined its effectiveness.Keywords
This publication has 11 references indexed in Scilit:
- Log-based predictive maintenance2014
- Mining unstructured log files for recurrent fault diagnosis2011
- Network IDS alert classification with frequent itemset mining and data clustering2010
- Detecting large-scale system problems by mining console logs2009
- System log pre-processing to improve failure prediction2009
- Dynamic syslog mining for network failure monitoring2005
- A data clustering algorithm for mining patterns from event logs2004
- Bursty and hierarchical structure in streams2002
- Support-vector networksMachine Learning, 1995
- Objective Criteria for the Evaluation of Clustering MethodsJournal of the American Statistical Association, 1971