LogDLR: Unsupervised Cross-System Log Anomaly Detection Through Domain-Invariant Latent Representation
- 5 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Dependable and Secure Computing
- Vol. 22 (4) , 4456-4471
Abstract
Log anomaly detection aims to discover abnormal events from massive log data to ensure the security and reliability of software systems. However, due to the heterogeneity of log formats and syntaxes across different systems, existing log anomaly detection methods often need to be designed and trained for specific systems, lacking generalization ability. To address this challenge, we propose LogDLR, a novel unsupervised cross-system log anomaly detection method. The core idea of LogDLR is to use universal sentence embeddings and a Transformer-based autoencoder to extract domain-invariant latent representations from log entries, which can effectively adapt to log format changes and capture semantic information and dependencies in log sequences. To obtain domain-invariant latent representations, we adopt a domain-adversarial training strategy, introducing a domain discriminator that competes with the Transformer-based encoder through a gradient reversal layer, forcing the encoder to learn shared knowledge between different system logs. Finally, the Transformer-based decoder detects anomalies based on the domain-invariant representations obtained by the encoder. We evaluate LogDLR in simulated cross-system scenarios using three publicly available log datasets. The experimental results show that LogDLR can handle heterogeneous logs effectively in cross-system scenarios and achieve efficient and accurate anomaly detection on both source and target systems.Keywords
Funding Information
- National Key Research and Development Program of China (2022YFB3104001)
- Key Research and Development Program of Hubei Province (2022BAA050)
- NSF (CNS2019340)
This publication has 43 references indexed in Scilit:
- AutoLog: Anomaly detection by deep autoencoding of system logsExpert Systems with Applications, 2022
- Log-based Anomaly Detection Without Log Parsing2021
- FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation2021
- Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection2019
- Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network2018
- Domain-Adversarial Training of Neural Networks2017
- Adversarial Discriminative Domain Adaptation2017
- A theory of learning from different domainsMachine Learning, 2009
- What Supercomputers Say: A Study of Five System Logs2007
- Failure diagnosis using decision trees2004