AutoLAD: An Unsupervised Anomaly Detection Method for Supercomputer Logs
Open Access
- 17 November 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 34-37
Abstract
Ensuring the reliability of the supercomputer is challenging. A potential solution could be to detect anomalies in a timely manner from the logs. However, current methods struggle with the vast diversity and volume of Exascale supercomputer logs due to the heavy reliance on manual labeling. This paper introduces AutoLAD, an unsupervised log anomaly detection method. Firstly, logs are embedded using a pre-trained language model. AutoLAD then creates a basic set of standard log patterns utilizing the Local Outlier Factor algorithm to remove anomalous entries. Finally, AutoLAD employs a single-log sliding-window strategy to handle the standard set, forming the autoencoder’s training set, which enables the learning of standard patterns. In the detection stage, the autoencoder evaluates a fixed-length log sequence. If the reconstruction error exceeds a set threshold, the sequence is deemed anomalous. Our approach, tested on logs from the NG-Tianhe supercomputer, achieved an F1 score of more than 0.91.Keywords
Funding Information
- China?s National Key Research and Development Program (2021YFB0300105)
This publication has 15 references indexed in Scilit:
- Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation2021
- Unsupervised log message anomaly detectionICT Express, 2020
- Robust log-based anomaly detection on unstable log data2019
- Doomsday: Predicting Which Node Will Fail When on Supercomputers2018
- GUIDE2017
- DeepLog2017
- Experience Report: Log Mining Using Natural Language Processing and Application to Anomaly Detection2017
- Incremental Clustering for Semi-Supervised Anomaly Detection applied on Log Data2017
- Experience Report: System Log Analysis for Anomaly Detection2016
- Co-analysis of RAS Log and Job Log on Blue Gene/P2011