OneLog: towards end-to-end software log anomaly detection
Open Access
- 16 April 2024
- journal article
- research article
- Published by Springer Nature in Automated Software Engineering
- Vol. 31 (2) , 1-36
Abstract
With the growth of online services, IoT devices, and DevOps-oriented software development, software log anomaly detection is becoming increasingly important. Prior works mainly follow a traditional four-staged architecture (Preprocessor, Parser, Vectorizer, and Classifier). This paper proposes OneLog, which utilizes a single deep neural network instead of multiple separate components. OneLog harnesses convolutional neural network (CNN) at the character level to take digits, numbers, and punctuations, which were removed in prior works, into account alongside the main natural language text. We evaluate our approach in six message- and sequence-based data sets: HDFS, Hadoop, BGL, Thunderbird, Spirit, and Liberty. We experiment with Onelog with single-, multi-, and cross-project setups. Onelog offers state-of-the-art performance in our datasets. Onelog can utilize multi-project datasets simultaneously during training, which suggests our model can generalize between datasets. Multi-project training also improves Onelog performance making it ideal when limited training data is available for an individual project. We also found that cross-project anomaly detection is possible with a single project pair (Liberty and Spirit). Analysis of model internals shows that one log has multiple modes of detecting anomalies and that the model learns manually validated parsing rules for the log messages. We conclude that character-based CNNs are a promising approach toward end-to-end learning in log anomaly detection. They offer good performance and generalization over multiple datasets. We will make our scripts publicly available upon the acceptance of this paper.Keywords
All Related Versions
Funding Information
- Research Council of Finland (298020 and 328058, 298020 and 328058)
- University of Oulu
This publication has 24 references indexed in Scilit:
- SiaLog: detecting anomalies in software execution logs using the siamese networkAutomated Software Engineering, 2022
- Log-based Anomaly Detection Without Log Parsing2021
- Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs2020
- Cluster Quality Analysis Using Silhouette Score2020
- Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network2018
- Spell: Streaming Parsing of System Event Logs2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Performance Evaluation of the Silhouette Index2015
- Experimentation in Software Engineering2012
- What Supercomputers Say: A Study of Five System Logs2007