Anomaly localization in large-scale clusters
Authors:
- 1 January 2007
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15525244,p. 322-330
Abstract
A critical problem facing by managing large-scale clusters is to identify the location of problems in a system in case of unusual events. As the scale of high performance computing (HPC) grows, systems are getting bigger. When a system fails to function properly, health-related data are collected for troubleshooting. However, due to the massive quantities of information obtained from a large number of components, the root causes of anomalies are often buried like needles in a haystack. In this paper, we present a localization method to automatically find out the potential root causes (i.e. a subset of nodes) of the problem from the overwhelming amount of data collected system-wide. System managers can focus on examining these potential locations, thereby significantly reducing human efforts required for anomaly localization. Our method consists of three interrelated steps: (1) feature collection to assemble a feature space for the system; (2) feature extraction to obtain the most significant features for efficient data analysis by applying the principal component analysis (PCA) algorithm; and (3) outlier detection to quickly identify the nodes that are ldquofar awayrdquo from the majority by using the cell-based detection algorithm. Preliminary studies are presented to demonstrate the potential of our method for localizing anomalies in a computing environment where the nodes perform comparable tasks.Keywords
This publication has 19 references indexed in Scilit:
- Diagnosing network-wide traffic anomalies2004
- Failure diagnosis using decision trees2004
- Critical event prediction for proactive management in large-scale computer clusters2003
- Predicting rare events in temporal domains2003
- Understanding large system failures-a fault injection experiment2003
- A statistical approach to predictive detectionComputer Networks, 2001
- Nonlinear Dimensionality Reduction by Locally Linear EmbeddingScience, 2000
- Distance-based outliers: algorithms and applicationsThe VLDB Journal, 2000
- The AppLeS Parameter Sweep Template: User-Level Middleware for the Grid2000
- Multicommodity flow models, failure propagation, and reliable loss network designIEEE/ACM Transactions on Networking, 1998