Supervised Learning for Detecting Cognitive Security Anomalies in Real-Time Log Data
Author:
- 17 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 1, 840-845
Abstract
Every system generates a large quantity of logs. Logs are immensely crucial for monitoring a system, inspecting anomalous behaviors, and analyzing errors. By using log data, many recent studies suggest that efficient and accurate machine learning classifiers can use to detect anomalies. The source diversification and the unstructured nature of logs create difficulties in subsequent analysis. Even though the evaluation of automatic log parsing opened up the door to further research. To decrease manual work, many anomaly detection systems based on automated log analysis have been developed. However, due to the lack of a research and comparison of multiple anomaly detection methods, developers may still be unsure about which anomaly detection methods to utilize. Even if developers employ an anomaly detection technique, reimplementation takes time. To address these issues, we present a comprehensive study of detecting security anomalies from real-time log data based on different supervised machine learning algorithms and trained with publicly archived logs data. Two selected production log datasets with 15,923,592 and 365,298 log records were used to evaluate these algorithms. We also proposed an approach that the ability to provide visibility into the system, the proper way to acquire insight using a strong monitoring system that collects metrics, represents data, and automated methods to give notifications to administrators to bring to their awareness a significant change in the system's status. We assessed the model's performance using a variety of well-known assessment metrics, including “precision”, “recall”, “specificity”, “F1 measure”, and “accuracy”.Keywords
This publication has 12 references indexed in Scilit:
- The Impact of Software Fault Prediction in Real-World Application2020
- An Evaluation Study on Log Parsing and Its Use in Log Mining2016
- Experience Report: System Log Analysis for Anomaly Detection2016
- Log clustering based problem identification for online service systems2016
- Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data2015
- Toward developing a systematic approach to generate benchmark datasets for intrusion detectionComputers & Security, 2012
- Detecting large-scale system problems by mining console logs2009
- A comparison of approaches to large-scale data analysis2009
- What Supercomputers Say: A Study of Five System Logs2007
- Analysis of Software Aging in a Web ServerIEEE Transactions on Reliability, 2006