Improving Performances of Log Mining for Anomaly Prediction Through NLP-Based Log Parsing
Authors:
- Nicolas AusselNANicolas Aussel
- Yohan PetetinYPYohan Petetin
- Sophie ChabridonSCSophie Chabridon
- 1 September 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 237-243
Abstract
Failure prediction of industrial systems is a promising application domain for data mining approaches and should naturally rely on log messages which are a prime source of data as they are generated by many systems. However, before extracting relevant information of such log messages, another critical step is to parse the logs, that is to say to transform a raw unstructured text from the log messages into a suitable input for data mining. These two problems (log parsing then log mining) are often studied separately while they are directly related in the context of failure prediction; moreover, few performance benchmarks are publicly available. In this paper, we focus on the impact of log parsing techniques via natural language processing on the performances of log mining on two datasets. The first one is a log of an industrial aeronautical system comprising over 4,500,000 messages collected over one year of operation; the second one is a public benchmark set from an HDFS cluster. On the latter, we show that it is possible to raise the F-score from 96% to 99.2% while using simpler and more robust log parsing techniques that require less parameter tuning provided that they are correctly combined with log mining techniques.Keywords
This publication has 11 references indexed in Scilit:
- DeepLog2017
- Holistic Processing and Exploring Event Logs2017
- Drain: An Online Log Parsing Approach with Fixed Depth Tree2017
- An Evaluation Study on Log Parsing and Its Use in Log Mining2016
- Log-based predictive maintenance2014
- Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised LearningFoundations and Trends in Computer Graphics and Vision, 2012
- A Lightweight Algorithm for Message Type Extraction in System Application LogsIEEE Transactions on Knowledge and Data Engineering, 2011
- Detecting large-scale system problems by mining console logs2009
- NLTK2002
- Support-vector networksMachine Learning, 1995