Taming of the Shrew: Modeling the Normal and Faulty Behaviour of Large-scale HPC Systems
Authors:
- Ana GainaruAGAna Gainaru
- Franck CappelloFCFranck Cappello
- William KramerWKWilliam Kramer
- 1 May 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15302075,p. 1168-1179
Abstract
HPC systems are complex machines that generate a huge volume of system state data called "events". Events are generated without following a general consistent rule and different hardware and software components of such systems have different failure rates. Distinguishing between normal system behaviour and faulty situation relies on event analysis. Being able to detect quickly deviations from normality is essential for system administration and is the foundation of fault prediction. As HPC systems continue to grow in size and complexity, mining event flows become more challenging and with the upcoming 10 Pet flop systems, there is a lot of interest in this topic. Current event mining approaches do not take into consideration the specific behaviour of each type of events and as a consequence, fail to analyze them according to their characteristics. In this paper we propose a novel way of characterizing the normal and faulty behaviour of the system by using signal analysis concepts. All analysis modules create ELSA (Event Log Signal Analyzer), a toolkit that has the purpose of modelling the normal flow of each state event during a HPC system lifetime, and how it is affected when a failure hits the system. We show that these extracted models provide an accurate view of the system output, which improves the effectiveness of proactive fault tolerance algorithms. Specifically, we implemented a filtering algorithm and short-term fault prediction methodology based on the extracted model and test it against real failure traces from a large-scale system. We show that by analyzing each event according to its specific behaviour, we get a more realistic overview of the entire system.Keywords
This publication has 18 references indexed in Scilit:
- Predicting Node Failure in High Performance Computing Systems from Failure and Usage Logs2011
- Event Log Mining Tool for Large Scale HPC Systems2011
- Removal of Correlated Noise by Modeling the Signal of Interest in the Wavelet DomainIEEE Transactions on Image Processing, 2009
- Blue Gene/L Log Analysis and Time to Interrupt Estimation2009
- What Supercomputers Say: A Study of Five System Logs2007
- A large-scale study of failures in high-performance computing systems2006
- An efficient methodology for noise characterization2005
- Automatic recognition of intermittent failures: an experimental study of field dataIEEE Transactions on Computers, 1990
- Bandpass Signal Sampling and Coherent DetectionIEEE Transactions on Aerospace and Electronic Systems, 1982
- The robustness of the two—samplet—test over the Pearson systemJournal of Statistical Computation and Simulation, 1978