Establishing Hypothesis for Recurrent System Failures from Cluster Log Files
Authors:
- Edward ChuahECEdward Chuah
- Gary LeeGLGary Lee
- William-Chandra TjhiWTWilliam-Chandra Tjhi
- Shyh-Hao KuoSKShyh-Hao Kuo
- Terence HungTHTerence Hung
- John HammondJHJohn Hammond
- Tommy MinyardTMTommy Minyard
- James C. BrowneJBJames C. Browne
- 1 December 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 15-22
Abstract
A goal for the analysis of supercomputer logs is to establish causal relationships among events which reflect significant state changes in the system. Establishing these relationships is at the heart of failure diagnosis. In principle, a log analysis tool could automate many of the manual steps systems administrators must currently use to diagnose system failures. However, supercomputer logs are unstructured, incomplete and contain considerable ambiguity so that direct discovery of causal relationships is difficult. This paper describes the second generation FDiag log-based failure diagnostics framework that provides automation of the manual failure diagnosis process and determines with high confidence, the likely cause of the failure, the components involved and the event sequences which contain the times of the causal and terminal events. FDiag extracts relevant events from the system logs, performs correlation analysis on these events and from these correlations determines the components involved and the event sequences. The diagnostics capabilities of FDiag are validated by comparing its assessments on known instances of recurrent failures on the Ranger supercomputer at the University of Texas at Austin. We believe FDiag is the first log analyzer to demonstrate this level of diagnostics capability from the system logs of an open source software stack incorporating Linux and the Lustre file system. FDiag will be put into production use for support of failure diagnosis on Ranger in September, 2011.Keywords
This publication has 19 references indexed in Scilit:
- Diagnosing the root-causes of failures from cluster log files2010
- End-to-end framework for fault management for open source clusters2010
- A study of dynamic meta-learning for failure prediction in large-scale systemsJournal of Parallel and Distributed Computing, 2010
- A practical failure prediction with location and lead time for Blue Gene/P2010
- System log pre-processing to improve failure prediction2009
- Toward Automated Anomaly Identification in Large-Scale SystemsIEEE Transactions on Parallel and Distributed Systems, 2009
- Exploring event correlation for failure prediction in coalitions of clusters2007
- Failure Prediction in IBM BlueGene/L Event Logs2007
- Pinpoint: problem determination in large, dynamic Internet services2003
- Models for time coalescence in event logs2003