Brain: Log Parsing With Bidirectional Parallel Tree
Top Cited Papers
- 26 April 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Services Computing
- Vol. 16 (5) , 3224-3237
Abstract
Automated log analysis can facilitate failure diagnosis for developers and operators using a large volume of logs. Log parsing is a prerequisite step for automated log analysis, which parses semi-structured logs into structured logs. However, existing parsers are difficult to apply to software-intensive systems, due to their unstable parsing accuracy on various software. Although neural network-based approaches are stable, their inefficiency makes it challenging to keep up with the speed of log production.We found that a logging statement always generate the same template words, thus, the word with the most frequency in each log is more likely to be constant. However, the identical constant and variable generated from different logging statements may break this rule Inspired by this key insight, we propose a new stable log parsing approach, called Brain, which creates initial groups according to the longest common pattern. Then a bidirectional tree is used to hierarchically complement the constant words to the longest common pattern to form the complete log template efficiently. Experimental results on 16 benchmark datasets show that our approach outperforms the state-of-the-art parsers on two widely-used parsing accuracy metrics, and it only takes around 46 seconds to process one million lines of logs.Keywords
Funding Information
- National Natural Science Foundation of China (62162003, 62102340, 61762008)
This publication has 29 references indexed in Scilit:
- DeepLog2017
- LogSed: Anomaly Diagnosis through Mining Time-Weighted Control Flow Graph in Logs2017
- Drain: An Online Log Parsing Approach with Fixed Depth Tree2017
- Syslog processing for switch failure diagnosis and prediction in datacenter networks2017
- Spell: Streaming Parsing of System Event Logs2016
- An Evaluation Study on Log Parsing and Its Use in Log Mining2016
- Toward Fine-Grained, Unsupervised, Scalable Performance Diagnosis for Production Cloud Computing SystemsIEEE Transactions on Parallel and Distributed Systems, 2013
- Abstracting log lines to log event types for mining software system logs2010
- Detecting large-scale system problems by mining console logs2009
- Abstracting Execution Logs to Execution Events for Enterprise Applications (Short Paper)2008