Model-based fault localization in large-scale computing systems
Authors:
- 1 April 2008
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15302075,p. 1-12
Abstract
We propose a new fault localization technique for software bugs in large-scale computing systems. Our technique always collects per-process function call traces of a target system, and derives a concise execution model that reflects its normal function calling behaviors using the traces. To find the cause of a failure, we compare the derived model with the traces collected when the system failed, and compute a suspect score that quantifies how likely a particular part of call traces explains the failure. The execution model consists of a call probability of each function in the system that we estimate using the normal traces. Functions with low probabilities in the model give high anomaly scores when called upon a failure. Frequently-called functions in the model also give high scores when not called. Finally, we report the function call sequences ranked with the suspect scores to the human analyst, narrowing further manual localization down to a small part of the overall system. We have applied our proposed method to fault localization of a known non-deterministic bug in a distributed parallel job manager. Experimental results on a three-site, 78-node distributed environment demonstrate that our method quickly locates an anomalous event that is highly correlated with the bug, indicating the effectiveness of our approach.Keywords
This publication has 12 references indexed in Scilit:
- Automatic Trace-Based Performance Analysis of Metacomputing Applications2007
- Anomaly localization in large-scale clusters2007
- Problem Diagnosis in Large-Scale Computing Environments2006
- On-line automated performance diagnosis on thousands of processes2006
- Detecting Application-Level Failures in Component-Based Internet ServicesIEEE Transactions on Neural Networks, 2005
- MRNet2003
- Performance debugging for distributed systems of black boxes2003
- An API for Runtime Code PatchingThe International Journal of High Performance Computing Applications, 2000
- A high-performance, portable implementation of the MPI message passing interface standardParallel Computing, 1996
- The Paradyn parallel performance measurement toolComputer, 1995