FP‐JSC: Job failure prediction on supercomputers through job application sequence correlation
- 24 March 2024
- journal article
- research article
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 36 (14) , e8041
Abstract
Summary: Supercomputers are advanced computing systems interconnected through high‐speed communication networks, consisting of independent computational nodes. During the unfolding of the big data era, the potent computational capabilities of these supercomputers play a pivotal role in scientific computing. Despite executing numerous advanced computational science and engineering tasks on supercomputers, many submitted jobs fail due to various factors, resulting in user inefficiencies. These failures not only consume system resources but also reduce the overall efficiency of the system. Previous research often couples job performance features with a single machine learning method for predicting job failure. However, a primary hurdle emerges from the high cost of gathering these features, complicating their real‐world applicability. To address this challenge, our study establishes correlations among job applications through extensive job log analysis. Leveraging correlations, we propose a predictive framework based on job application sequence correlation (called FP‐JSC). This innovative framework employs multiple machine learning models to offer holistic predictions, selecting the most suitable model based on its learning effectiveness. Moreover, the framework optimizes feature collection expenses without adversely affecting job execution. We determine job applications using both job paths and job names, with the former emerging as a novel feature derived from supplementary monitoring data. Empirical results underscore FP‐JSC's effectiveness, accurately identifying over 89% of jobs with 95% specificity and 89% sensitivity—outperforming single prediction methods employed in related works.Keywords
Funding Information
- National Natural Science Foundation of China (62202471)
This publication has 25 references indexed in Scilit:
- PreF: Predicting job failure on supercomputers with job path and user behaviorConcurrency and Computation: Practice and Experience, 2022
- Failure prediction using machine learning in a virtualised HPC system and applicationCluster Computing, 2019
- Cost-effective and fault-resilient reusability prediction model by using adaptive genetic algorithm based neural network for web-of-service applicationsCluster Computing, 2018
- Software reliability modeling using increased failure interval with ANNCluster Computing, 2018
- Hora: Architecture-aware online failure predictionJournal of Systems and Software, 2018
- Software defect prediction techniques using metrics based on neural network classifierCluster Computing, 2018
- Deep neural network based hybrid approach for software defect prediction using software metricsCluster Computing, 2018
- Scalable system scheduling for HPC and big dataJournal of Parallel and Distributed Computing, 2018
- Failure Prediction Methodology for Improved Proactive Maintenance using Bayesian Approach ★ ★The authors gratefully acknowledge STMicroelectronics for their support and provision of data for TT case study. The authors also acknowledge European project INTEGRATE and region RhoneAlpes for ongoing Research.IFAC-PapersOnLine, 2015
- SLURM: Simple Linux Utility for Resource Management2003