Improving HPC System Performance by Predicting Job Resources via Supervised Machine Learning
Authors:
Open Access
- 28 July 2019
- conference paper
- Published by Association for Computing Machinery (ACM)
- Vol. 2019, 1-8
Abstract
High-Performance Computing (HPC) systems are resources utilized for data capture, sharing, and analysis. The majority of our HPC users come from other disciplines than Computer Science. HPC users including computer scientists have difficulties and do not feel proficient enough to decide the required amount of resources for their submitted jobs on the cluster. Consequently, users are encouraged to over-estimate resources for their submitted jobs, so their jobs will not be killing due insufficient resources. This process will waste and devour HPC resources; hence, this will lead to inefficient cluster utilization. We created a supervised machine learning model and integrated it into the Slurm resource manager simulator to predict the amount of required memory resources (Memory) and the required amount of time to run the computation. Our model involves using different machine learning algorithms. Our goal is to integrate and test the proposed supervised machine learning model on Slurm. We used over 10000 tasks selected from our HPC log files to evaluate the performance and the accuracy of our integrated model. The purpose of our work is to increase the performance of the Slurm by predicting the amount of require jobs memory resources and the time required for each particular job in order to improve the utilization of the HPC system using our integrated supervised machine learning model. Our results indicate that for larger jobs our model helps dramatically reduce computational turnaround time (from five days to ten hours for large jobs), substantially increased utilization of the HPC system, and decreased the average waiting time for the submitted jobs.Keywords
Funding Information
- National Science Foundation (CNS-1006860, EPS-1006860, EPS-0919443, ACI-1440548, CHE-1726332)
- National Institutes of Health (NIH P20GM113109)
This publication has 7 references indexed in Scilit:
- A Slurm Simulator: Implementation and Parametric Analysis2017
- Prediction and characterization of application power use in a high‐performance computing environmentStatistical Analysis and Data Mining: An ASA Data Science Journal, 2017
- Improving backfilling by using machine learning to predict running times2015
- Advanced Geoinformation Science2010
- Towards energy-aware scheduling in data centers using machine learning2010
- On the Use of Machine Learning to Predict the Time and Resources Consumed by Applications2010
- Machine learning: a review of classification and combining techniquesArtificial Intelligence Review, 2006