Queue Waiting Time Prediction for Large-scale High-performance Computing System
Author:
- 1 July 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 850-855
Abstract
Traditionally, high-performance computing (HPC) systems have been extensively utilized in many science fields including big data analysis and machine learning. Such large-scale HPC resources typically use the queue management systems which prefer space-sharing method to allocate resources. In space-sharing method, it is natural that a queue waiting time occurs until the resources are available if resources are not sufficient. When the predicted information on such a waiting time is provided, it is possible to improve the performance of scheduler. For this, in this paper, we propose a prediction method of queue waiting time based on the job log file created in the HPC system actually under operation. The prediction technique presented in this paper largely comprises three phases. The first phase is a pre-processing of data in constant time intervals. In the second phase, major features are selected through a factor analysis and clustering is conducted based on the selected features. In the third phase, a waiting time of the next job is predicted using the sliding window method based on the jobs that were clustered.Keywords
This publication has 17 references indexed in Scilit:
- Sliding Windows and Persistence: An Application of Topological Methods to Signal AnalysisFoundations of Computational Mathematics, 2014
- Survival analysis and regression modelsJournal of Nuclear Cardiology, 2014
- PBS Pro: Grid Computing and Scheduling Attributes2004
- An integrated approach to parallel scheduling using gang-scheduling, backfilling, and migrationIEEE Transactions on Parallel and Distributed Systems, 2003
- SLURM: Simple Linux Utility for Resource Management2003
- Using Run-Time Predictions to Estimate Queue Wait Times and Improve Scheduler Performance1999
- Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization1999
- A multivariate Kolmogorov-Smirnov test of goodness of fitStatistics & Probability Letters, 1997
- Using queue time predictions for processor allocation1997
- The ANL/IBM SP scheduling system1995