A Survey of Log-Correlation Tools for Failure Diagnosis and Prediction in Cluster Systems
Open Access
- 21 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 133487-133503
Abstract
System logs are the first source of information available to system designers to analyze and troubleshoot their cluster systems. For example, High-Performance Computing (HPC) systems generate a large volume of heterogeneous data from multiple sub-systems, so the idea of using a single source of data to achieve a given goal, such as identification of failures, is losing its validity. System log-analysis tools assist system designers gain understanding into a large volume of system logs. They enable system designers to perform various analyses (e.g., diagnosing node failures or predicting node failures). Current system log-analysis tools vary significantly in their function and design.We conduct a systematic review of literature on system log-analysis tools and select 46 representative articles out of 3,758 initial articles. To the best of our knowledge, there is no work that studied the characteristics of log-correlation tools (LogCTs) with respect to four quality attributes including (a) spurious correlations, (b) correlation threshold settings, (c) outliers in the data and (d) missing data. In this paper, we (a) propose a quality model to evaluate LogCTs and (b) use this quality model to evaluate and recommend current LogCTs. Through our review, we (a) identify papers on LogCTs, (b) build a quality model consisting of the four quality attributes and (c) discuss several open challenges for future research. Our study highlights the advantages and limitations of existing LogCTs and identifies research opportunities that could facilitate better failure handling in large cluster systems.Keywords
Funding Information
- Auspices of the European Union (EU) Horizon 2020 Research and Innovation Program [Cyber security competence for Research and innovation (CONCORDIA)] (830927)
- Security Lancaster under the Engineering and Physical Sciences Research Council (EP/V026763/1)
This publication has 45 references indexed in Scilit:
- A Principled Approach to HPC Event Monitoring2015
- The instability of the Pearson correlation coefficient in the presence of coincidental outliersFinance Research Letters, 2015
- Digging deeper into cluster system logs for failure prediction and root cause diagnosis2014
- Comprehensive, open‐source resource usage measurement and analysis for HPC systemsConcurrency and Computation: Practice and Experience, 2014
- Failure Diagnosis of Complex Systems2012
- Baler: deterministic, lossless log message clustering toolComputer Science - Research and Development, 2011
- A study of dynamic meta-learning for failure prediction in large-scale systemsJournal of Parallel and Distributed Computing, 2010
- A survey of online failure prediction methodsACM Computing Surveys, 2010
- MISSING DATA: A CONCEPTUAL REVIEW FOR APPLIED PSYCHOLOGISTSPersonnel Psychology, 1994
- The Computation of Partial Correlation CoefficientsJournal of the American Statistical Association, 1946