A Semisupervised Approach for Industrial Anomaly Detection via Self-Adaptive Clustering
- 29 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 20 (2) , 1687-1697
Abstract
With the rapid development of the Industrial Internet of Things, log-based anomaly detection has become vital for smart industrial construction that has prompted many researchers to contribute. To detect anomalies based on log data, semisupervised approaches stand out from supervised and unsupervised approaches because they only require a portion of labeled data and are relatively stable. However, the state-of-the-art semisupervised approaches still suffer from two main problems: manual parameter setting and unsatisfactory performance with high false positives. We propose AdaLog, an integrated semisupervised approach based on self-adaptive clustering, for industrial anomaly detection. In particular, the clustering step performs automatic label probability estimation by distinguishing 12 situations so that the label probability of each unlabeled data can be carefully calculated, leading to high accuracy. In addition, AdaLog employs a pretrained model to learn contextual information comprehensively and a transformer-based model to detect anomalies efficiently. To alleviate class imbalance, an undersampling method is incorporated. The results on three popular datasets demonstrate that AdaLog significantly outperforms three state-of-the-art semisupervised approaches by 17.8%–2489.8% on average in terms of F1-score, and is even superior to two supervised approaches in most cases with average improvements of 10.9%–23.8%.Keywords
Funding Information
- General Research Fund of the Research Grants Council of Hong Kong (11208017)
- City University of Hong Kong (7005028, 7005217, 6000796)
- other industry projects (9229109, 9229098, 9229029, 9220103, 9220097, 9440227)
This publication has 28 references indexed in Scilit:
- Integrated Generative Model for Industrial Anomaly Detection via Bidirectional LSTM and Attention MechanismIEEE Transactions on Industrial Informatics, 2021
- Determine the Number of Unknown Targets in Open World Based on Elbow MethodIEEE Transactions on Fuzzy Systems, 2020
- Anomaly Detection for IoT Time-Series Data: A SurveyIEEE Internet of Things Journal, 2019
- LSTM Learning With Bayesian and Gaussian Processing for Anomaly Detection in Industrial IoTIEEE Transactions on Industrial Informatics, 2019
- Security Log Analysis in Critical Industrial Systems Exploiting Game Theoretic Feature Selection and Evidence CombinationIEEE Transactions on Industrial Informatics, 2019
- Robust log-based anomaly detection on unstable log data2019
- DeepLog2017
- Incremental Clustering for Semi-Supervised Anomaly Detection applied on Log Data2017
- Learning from Imbalanced DataIEEE Transactions on Knowledge and Data Engineering, 2009
- Failure Prediction in IBM BlueGene/L Event Logs2007