Toward Automated Anomaly Identification in Large-Scale Systems
Authors:
- 17 April 2009
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 21 (2) , 174-187
Abstract
When a system fails to function properly, health-related data are collected for troubleshooting. However, it is challenging to effectively identify anomalies from the voluminous amount of noisy, high-dimensional data. The traditional manual approach is time-consuming, error-prone, and even worse, not scalable. In this paper, we present an automated mechanism for node-level anomaly identification in large-scale systems. A set of techniques is presented to automatically analyze collected data: data transformation to construct a uniform data format for data analysis, feature extraction to reduce data size, and unsupervised learning to detect the nodes acting differently from others. Moreover, we compare two techniques, principal component analysis (PCA) and independent component analysis (ICA), for feature extraction. We evaluate our prototype implementation by injecting a variety of faults into a production system at NCSA. The results show that our mechanism, in particular, the one using ICA-based feature extraction, can effectively identify faulty nodes with high accuracy and low computation overhead.Keywords
This publication has 39 references indexed in Scilit:
- Exploring event correlation for failure prediction in coalitions of clusters2007
- A survey of fault localization techniques in computer networksScience of Computer Programming, 2004
- Basic concepts and taxonomy of dependable and secure computingIEEE Transactions on Dependable and Secure Computing, 2004
- Failure diagnosis using decision trees2004
- Evolution of multivariate statistical process control: application of independent component analysis and external analysisComputers & Chemical Engineering, 2004
- Face recognition by independent component analysisIEEE Transactions on Neural Networks, 2002
- A statistical approach to predictive detectionComputer Networks, 2001
- Distance-based outliers: algorithms and applicationsThe VLDB Journal, 2000
- Multicommodity flow models, failure propagation, and reliable loss network designIEEE/ACM Transactions on Networking, 1998
- Fault injection experiments using FIATIEEE Transactions on Computers, 1990