Promoting Resource Utilization in HPC via Scheduling
- 19 November 2025
- journal article
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 37 (27-28)
Abstract
The increasing complexity of supercomputing workloads poses challenges to efficient resource management, especially in balancing computational and I/O demands. Shared burst buffers, as high‐speed intermediate storage, offer a promising avenue to mitigate I/O bottlenecks. However, naive job scheduling strategies often neglect the potential of burst buffers, relying on heuristic methods with limited adaptability. To address this, we propose an innovative burst‐buffer‐aware scheduling framework that integrates burst buffer capacity into job scheduling as a key resource. Through multi‐objective optimization, the framework intelligently balances trade‐offs among job waiting time, slowdown, and completion time, surpassing the rigidity of conventional approaches. Leveraging real‐world workload traces, the framework dynamically adapts to varying windows and job characteristics, combining computation and burst buffer demands to optimize scheduling decisions. Experimental results reveal that the proposed framework enhances scheduling efficiency and system adaptability, establishing a smarter and more effective approach to supercomputing job scheduling. This work underscores the importance of burst‐buffer‐aware strategies in advancing high‐performance computing, offering novel insights into intelligent resource management.Keywords
Funding Information
- National Natural Science Foundation of China (62202471)
This publication has 30 references indexed in Scilit:
- Mobilizing underutilized storage nodes via job path: A job-aware file striping approachParallel Computing, 2024
- Deep reinforcement learning-based methods for resource scheduling in cloud computing: a review and future directionsArtificial Intelligence Review, 2024
- DRLBTSA: Deep reinforcement learning based task-scheduling algorithm in cloud computingMultimedia Tools and Applications, 2023
- Analyzing Resource Utilization in an HPC System: A Case Study of NERSC’s Perlmutter2023
- An empirical study of I/O separation for burst buffers in HPC systemsJournal of Parallel and Distributed Computing, 2021
- Performance characterization of scientific workflows for the optimal use of Burst BuffersFuture Generation Computer Systems, 2020
- Evalix: Classification and Prediction of Job Resource Consumption on HPC Platforms2017
- Particle swarm optimization algorithm: an overviewSoft Computing, 2017
- Visualizing the Pareto Frontier2008
- Simulated Annealing2006