Anomaly Detection Using System Logs
- 12 November 2021
- journal article
- research article
- Published by IGI Global Scientific Publishing in International Journal of Information Security and Privacy
- Vol. 16 (1) , 1-15
Abstract
Anomaly detection is a very important step in building a secure and trustworthy system. Manually it is daunting to analyze and detect failures and anomalies. In this paper, we proposed an approach that leverages the pattern matching capabilities of Convolution Neural Network (CNN) for anomaly detection in system logs. Features from log files are extracted using a windowing technique. Based on this feature, a one-dimensional image (1×n dimension) is generated where the pixel values of an image correlate with the features of the logs. On these images, the 1D Convolution operation is applied followed by max pooling. Followed by Convolution layers, a multi-layer feed-forward neural network is used as a classifier that learns to classify the logs as normal or abnormal from the representation created by the convolution layers. The model learns the variation in log pattern for normal and abnormal behavior. The proposed approach achieved improved accuracy compared to existing approaches for anomaly detection in Hadoop Distributed File System (HDFS) logs.Keywords
This publication has 43 references indexed in Scilit:
- Online anomaly detection using dimensionality reduction techniques for HTTP log analysisComputer Networks, 2015
- Anomaly Detection from Log Files Using Data Mining Techniques2015
- A survey on unsupervised outlier detection in high‐dimensional numerical dataStatistical Analysis and Data Mining: An ASA Data Science Journal, 2012
- Isolation-Based Anomaly DetectionACM Transactions on Knowledge Discovery From Data, 2012
- Mining dependency in distributed systems through unstructured logs analysisACM SIGOPS Operating Systems Review, 2010
- Anomaly detectionACM Computing Surveys, 2009
- Feature bagging for outlier detection2005
- Performance debugging for distributed systems of black boxesACM SIGOPS Operating Systems Review, 2003
- Anomaly-based intrusion detection: privacy concerns and other problemsComputer Networks, 2000
- Theory of the Backpropagation Neural Network**Based on “nonindent” by Robert Hecht-Nielsen, which appeared in Proceedings of the International Joint Conference on Neural Networks 1, 593–611, June 1989. © 1989 IEEE.1992