Analyzing HPC Job Wait Times under Resource Scaling Using Historical Workload Data
Open Access
- 26 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Abstract
Understanding the impact of hardware configuration, infrastructure investments, and operational policies on job wait times in high-performance computing (HPC) systems is a challenging problem primarily due to the lack of effective tools that are built on controlled, real-world observations across different system configurations. In this work, we present a data-driven framework that leverages historical job traces to estimate the impact of resource modifications on queue performance. Using job data collected from multiple HPC systems during periods with no hardware changes, we construct a synthetic scheduler that replays real job submission patterns and resource requests, while using ground-truth runtimes to model execution. This enables controlled, counterfactual evaluation of infrastructure changes without modifying production systems. We perform experiments by scaling system resources, including GPUs and CPU cores. Our results on GPU-based systems show that increasing GPU capacity leads to significant reductions in wait times, while CPU-only scaling provides minimal benefit. To quantify these effects, we introduce the Weighted Wait-Time Score (WWS), a bounded metric that captures both typical and tail wait-time behavior. Our experiments show that informed resource scaling improves WWS (up to 92% relative gain in WWS for low-baseline systems), capturing gains in both typical and tail wait-time behavior under this metric. We further formulate a cost-aware optimization framework to guide resource allocation under budget constraints. Our approach offers a data-driven way to evaluate HPC upgrades and supports future predictive optimization and better resource use. Overall, this work provides (i) a trace-driven framework for evaluating resource scaling effects, (ii) a bounded metric (WWS) for comparing wait-time performance, and (iii) a pathway to optimal user wait time optimization based resource allocation in HPC systems.Keywords
This publication has 10 references indexed in Scilit:
- MetaPilot: A DRL-based controller for dynamic adaptation to shifting scheduling objectives in HPC systemsFuture Generation Computer Systems, 2025
- Predictive Modeling of HPC Job Queue Times: Improving User Decision-Making and Resource Utilization2025
- Tandem Predictions for HPC jobs2024
- Parallel Backfill: Improving HPC System Performance by Scheduling Jobs in Parallel2024
- A Reinforcement Learning Based Backfilling Strategy for HPC Batch Jobs2023
- Machine Learning Assisted HPC Workload Trace Generation for Leadership Scale Storage Systems2022
- Hybrid Workload Scheduling on HPC Systems2022
- In Datacenter Performance, The Only Constant Is Change2020
- Scheduling Beyond CPUs for HPC2019
- Open XDMoD: A Tool for the Comprehensive Management of High-Performance Computing ResourcesComputing in Science & Engineering, 2015