FlexParser—The adaptive log file parser for continuous results in a changing world
Authors:
Open Access
- 27 January 2022
- journal article
- research article
- Published by Wiley in Journal of Software: Evolution and Process
- Vol. 34 (3) , e2426
Abstract
Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi‐structured log messages. State‐of‐the‐art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes “mutation” and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel and flexible parser, calledFlexParser, which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several examples. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1‐Score of 0.98, it outperforms other Deep Learning methods as well as state‐of‐the‐art unsupervised parsers.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Deep learningNature, 2015
- Handwritten Word Spotting with Corrected Attributes2013
- LogSig2011
- An automated approach for abstracting execution logs to execution eventsJournal of Software Maintenance and Evolution: Research and Practice, 2008
- AUC: a misleading measure of the performance of predictive distribution modelsGlobal Ecology and Biogeography, 2007
- Dynamic syslog mining for network failure monitoring2005
- Mining Frequent Patterns without Candidate Generation: A Frequent-Pattern Tree ApproachData Mining and Knowledge Discovery, 2004
- Critical event prediction for proactive management in large-scale computer clusters2003
- A fast and elitist multiobjective genetic algorithm: NSGA-IIIEEE Transactions on Evolutionary Computation, 2002
- Long Short-Term MemoryNeural Computation, 1997