CRUDE: Combining Resource Usage Data and Error Logs for Accurate Error Detection in Large-Scale Distributed Systems
Authors:
- Nentawe GurumdimmaNGNentawe Gurumdimma
- Arshad JhumkaAJArshad Jhumka
- Maria LiakataMLMaria Liakata
- Edward ChuahECEdward Chuah
- James BrowneJBJames Browne
- 22 December 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 51-60
Abstract
The use of console logs for error detection in large scale distributed systems has proven to be useful to system administrators. However, such logs are typically redundant and incomplete, making accurate detection very difficult. In an attempt to increase this accuracy, we complement these incomplete console logs with resource usage data, which captures the resource utilisation of every job in the system. We then develop a novel error detection methodology, the CRUDE approach, that makes use of both the resource usage data and console logs. We thus make the following specific technical contributions: we develop (i) a clustering algorithm to group nodes with similar behaviour, (ii) an anomaly detection algorithm to identify jobs with anomalous resource usage, (iii) an algorithm that links jobs with anomalous resource usage with erroneous nodes. We then evaluate our approach using console logs and resource usage data from the Ranger Supercomputer. Our results are positive: (i) our approach detects errors with a true positive rate of about 80%, and (ii) when compared with the well-known Nodeinfo error detection algorithm, our algorithm provides an average improvement of around 85% over Nodeinfo, with a best-case improvement of 250%.Keywords
This publication has 18 references indexed in Scilit:
- Exploring void search for fault detection on extreme scale systems2014
- Fault prediction under the microscope: A closer look into HPC systems2012
- Anomaly Detection via Online Oversampling Principal Component AnalysisIEEE Transactions on Knowledge and Data Engineering, 2012
- Taming of the Shrew: Modeling the Normal and Faulty Behaviour of Large-scale HPC Systems2012
- System State Discovery Via Information Content Clustering of System Logs2011
- End-to-end framework for fault management for open source clusters2010
- Toward Automated Anomaly Identification in Large-Scale SystemsIEEE Transactions on Parallel and Distributed Systems, 2009
- Alert Detection in System Logs2008
- Mining anomalies using traffic feature distributionsACM SIGCOMM Computer Communication Review, 2005
- Basic concepts and taxonomy of dependable and secure computingIEEE Transactions on Dependable and Secure Computing, 2004