Runtime Performance Anomaly Diagnosis in Production HPC Systems Using Active Learning
- 13 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 35 (4) , 693-706
Abstract
With the increasing scale and complexity of High-Performance Computing (HPC) systems, performance variations in applications caused by anomalies have become significant bottlenecks in system health and operational efficiency. As we move towards exascale systems, these variations become more prominent due to the increased sharing of resources. Such variations lead to lower energy efficiency and higher operational costs. To mitigate these problems, one must quickly and accurately diagnose the root cause of the anomalies at scale. One way to evaluate system health and identify the underlying causes is by manually examining certain performance metrics in telemetry data or using rule-based methods. Due to the daily size of telemetry data reaching terabytes and the fact that the numeric telemetry data contains thousands of metrics, manual analysis of telemetry to diagnose problems becomes challenging. Given these limitations, Machine Learning (ML)-based approaches have been gaining popularity as they have been shown to be effective and practical in diagnosing previously encountered performance anomalies. One primary challenge for supervised ML models is that they require a significant amount of labeled samples during training. However, obtaining many labels for anomalies is extremely difficult and costly, considering anomalies occur infrequently and real-world numeric system telemetry data is hard to label since it contains thousands of metrics. This paper proposes a novel active learning-based framework that diagnoses performance anomalies (i.e., identifying the type of an anomaly) in HPC systems at runtime using significantly fewer labeled samples compared to state-of-the-art ML-based approaches. We show that the proposed framework achieves the same F1-score compared to a supervised approach using much fewer labeled samples (i.e., 16x fewer samples for achieving a 0.78 F1-score, 11x fewer samples for achieving a 0.82 F1-score), even when there are previously unseen applications and application inputs in the test dataset.Keywords
Funding Information
- Sandia National Laboratories
- National Technology and Engineering Solutions of Sandia, LLC.
- Honeywell International, Inc.
- U.S. Department of Energy's National Nuclear Security Administration (DE-NA0003525)
This publication has 42 references indexed in Scilit:
- E2EWatch: An End-to-End Anomaly Diagnosis Framework for Production HPC Systems2021
- Proctor: A Semi-Supervised Performance Anomaly Diagnosis Framework for Production HPC Systems2021
- Q-learning and LSTM based deep active learning strategy for malware defense in industrial IoT applicationsMultimedia Tools and Applications, 2021
- Improving security using SVM-based anomaly detection: issues and challengesSoft Computing, 2020
- FINJ: A Fault Injection Tool for HPC Systems2018
- Taxonomist: Application Detection Through Rich Monitoring Data2018
- Diagnosing Performance Variations in HPC Applications Using Machine Learning2017
- A Fourth Order Accurate Finite Difference Scheme for the Elastic Wave Equation in Second Order FormulationJournal of Scientific Computing, 2011
- Kullback-Leibler Divergence2011
- Fast Parallel Algorithms for Short-Range Molecular DynamicsJournal of Computational Physics, 1995