Doomsday: Predicting Which Node Will Fail When on Supercomputers
Authors:
- Anwesha DasADAnwesha Das
- Frank MuellerFMFrank Mueller
- Paul HargrovePHPaul Hargrove
- Eric RomanEREric Roman
- Scott BadenSBScott Baden
- 1 November 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 108-121
Abstract
Predicting which node will fail and how soon remains a challenge for HPC resilience, yet may pave the way to exploiting proactive remedies before jobs fail. Not only for increasing scalability up to exascale systems but even for contemporary supercomputer architectures does it require substantial efforts to distill anomalous events from noisy raw logs. To this end, we propose a novel phrase extraction mechanism called TBP (time-based phrases) to pin-point node failures, which is unprecedented. Our study, based on real system data and statistical machine learning, demonstrates the feasibility to predict which specific node will fail in Cray systems. TBP achieves no less than 83% recall rates with lead times as high as 2 minutes. This opens up the door for enhancing prediction lead times for supercomputing systems in general, thereby facilitating efficient usage of both computing capacity and power in large scale production systems.Keywords
This publication has 47 references indexed in Scilit:
- A Scalable, Non-Parametric Method for Detecting Performance Anomaly in Large Scale ComputingIEEE Transactions on Parallel and Distributed Systems, 2015
- LogDiver2015
- Digging deeper into cluster system logs for failure prediction and root cause diagnosis2014
- Lazy Checkpointing: Exploiting Temporal Locality in Failures to Mitigate Checkpointing Overheads on Extreme-Scale Systems2014
- Failure prediction for HPC systems and applicationsThe International Journal of High Performance Computing Applications, 2013
- Event Log Mining Tool for Large Scale HPC Systems2011
- A study of dynamic meta-learning for failure prediction in large-scale systemsJournal of Parallel and Distributed Computing, 2010
- A survey of online failure prediction methodsACM Computing Surveys, 2010
- An analysis of clustered failures on large supercomputing systemsJournal of Parallel and Distributed Computing, 2009
- 'Neural-gas' network for vector quantization and its application to time-series predictionIEEE Transactions on Neural Networks, 1993