Towards Universal Segmentation for Log Parsing
Open Access
- 12 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 207-219
Abstract
Log parsing is a crucial step in log analysis facilitating program comprehension throughout software maintenance and engineering life cycles. Log parsing transforms unstructured log messages into structured data required by various downstream analysis tasks. The sheer volume of log data generated by modern software systems motivates the development of numerous log parsing techniques in the literature. However, existing log parsers still suffer from unsatisfactory accuracy, which may significantly affect the follow-up analysis such as log-based anomaly detection. We have identified two main limitations that hinder the effectiveness of existing log parsing methods: (1) under-segmentation: most log parsers leverage a fixed, predefined set of delimiters to separate a log message into a set of tokens, which may fail to split log messages correctly due to the heterogeneity of logging formats; (2) over-segmentation: using too many delimiters may lead to the over-segmentation issue, which fragments meaningful units in log messages and makes it difficult to accurately identify templates and parameters. To address these limitations, we propose SCLog, a novel syntax- and contextual-aware segmentation approach for log parsing. SCLog leverages a comprehensive set of syntax-based heuristics to segment log messages into coarse-grained tokens. To further tokenize log messages into fine-grained tokens, SCLog mines the structural patterns of tokens based on their surrounding contexts to identify the optimal delimiters for each token dynamically. We evaluate SCLog on widely-used, large-scale Loghub-2.0 datasets. The results demonstrate that SCLog significantly improves the parsing accuracy of four representative log parsers.Keywords
Funding Information
- Luxembourg National Research Fund (FNR) (C22/IS/17373407/LOGODOR)
This publication has 32 references indexed in Scilit:
- Impact of log parsing on deep learning-based anomaly detectionEmpirical Software Engineering, 2024
- An empirical study of the impact of log parsers on the performance of log-based anomaly detectionEmpirical Software Engineering, 2022
- Log-based Anomaly Detection Without Log Parsing2021
- Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs2020
- Tools and Benchmarks for Automated Log Parsing2019
- Drain: An Online Log Parsing Approach with Fixed Depth Tree2017
- Spell: Streaming Parsing of System Event Logs2016
- An Evaluation Study on Log Parsing and Its Use in Log Mining2016
- LogCluster - A data clustering and pattern mining algorithm for event logs2015
- Detection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data2015