Adaptively Periodic I/O Scheduling for Concurrent HPC Applications
Open Access
- 21 April 2022
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 11 (9) , 1318
Abstract
With the convergence of big data and HPC (high-performance computing), various machine learning applications and traditional large-scale simulations with a stochastically iterative I/O periodicity are running concurrently on HPC platforms, which poses more challenges on the scarcely shared I/O resources due to the ever-growing data transfer demand. Currently the existing heuristic online and periodic offline I/O scheduling methods for traditional HPC applications with a fixed I/O periodicity are not suitable for the applications with stochastically iterative I/O periodicities, which are required to schedule the concurrent I/Os from different applications under I/O congestion. In this work, we propose an adaptively periodic I/O scheduling (APIO) method that optimizes the system efficiency and application dilation by taking the stochastically iterative I/O periodicity of the applications into account. We first build a periodic offline scheduling method within a specified duration to capture the iterative nature. After that, APIO adjusts the bandwidth allocation to resist stochasticity based on the actual length of the computing phrase. In the case where the specified duration does not satisfy the actual running requirements, the period length will be extended to adapt to the actual duration. Theoretical analysis and extensive simulations demonstrate the efficiency of our proposed I/O scheduling method over the existing online approach.Keywords
Funding Information
- Key-Area Research and Development Plan of Guangdong Province (#2020B010164003)
- National Key Research and Development Plan's Key Special Program on High performance computing of China (#2017YFB0203201)
This publication has 30 references indexed in Scilit:
- CARS: A contention-aware scheduler for efficient resource management of HPC storage systemsParallel Computing, 2019
- I/O Scheduling Strategy for Periodic ApplicationsACM Transactions on Parallel Computing, 2019
- Software-defined QoS for I/O in exascale computingCCF Transactions on High Performance Computing, 2019
- What Size Should Your Buffers to Disks be?2018
- A Checkpoint of Research on Parallel I/O for High-Performance ComputingACM Computing Surveys, 2018
- Periodic I/O Scheduling for Super-Computers2017
- Storage wall for exascale supercomputingFrontiers of Information Technology & Electronic Engineering, 2016
- AGIOS: Application-Guided I/O Scheduling for Parallel File Systems2013
- The HaLoop approach to large-scale iterative data analysisThe VLDB Journal, 2012
- The Fourth Paradigm: Data-Intensive Scientific Discovery [Point of View]Proceedings of the IEEE, 2011