Leveraging Pre-Built Catalogs and Object-Level Scheduling to Eliminate I/O Bottlenecks in HPC Environments
Open Access
- 20 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 55984-55995
Abstract
Modern High-Performance Computing (HPC) environments face mounting challenges due to the shift from large to small file datasets, combined with an increasing number of users and parallelized applications. As HPC systems rely on Parallel File Systems (PFS), such as Lustre for data processing, performance bottlenecks stemming from Object Storage Target (OST) contention have become a significant concern. Existing solutions, such as LADS with its object-level scheduling approach, fall short in large-scale HPC environments due to their inability to effectively address metadata I/O bottlenecks and the growing number of I/O processes. This study highlights the pressing need for a comprehensive solution that tackles both OST contention and metadata I/O challenges in diverse HPC workloads. To address these challenges, we propose SwiftLoad, an object-level I/O scheduling framework that leverages a metadata catalog to improve the performance and efficiency of parallel HPC utilities. The adoption of the metadata catalog mitigates the metadata I/O bottlenecks that commonly occur in HPC utilities, a challenge that is particularly pronounced in object-level I/O scheduling. SwiftLoad addresses OST contention and the uneven distribution of I/O processes across different OSTs through mathematical modeling and incorporates a Loader Configuration Module to regulate the number of I/O processes. Evaluated with two representative utilities—data deduplication profiling and data augmentation—SwiftLoad demonstrated performance improvements of up to 5.63× and 11.0×, respectively, on a production supercomputer.Keywords
Funding Information
- National Research Foundation of Korea
- Korean Government [Ministry of Science and ICT (MSIT)] (RS-2024-00416666)
- Korea Institute of Science and Technology Information (K25L2M2C2)
- Oak Ridge Leadership Computing Facility, located at the National Center for Computational Sciences at the Oak Ridge National Laboratory, which is supported by the Office of Science of the DOE (DE-AC05-00OR22725)
This publication has 10 references indexed in Scilit:
- Xfast: Extreme File Attribute Stat Acceleration for Lustre2023
- Hvac: Removing I/O Bottleneck for Large-Scale Deep Learning Applications2022
- LABIOS2019
- A Bloom Filter Based Scalable Data Integrity Check Tool for Large-Scale Dataset2016
- Distributed and Scalable Directory Service in a Parallel File SystemIEICE Transactions on Information and Systems, 2016
- Log-Less Metadata Management on Metadata Server for Parallel File SystemsThe Scientific World Journal, 2014
- Optimizing a hybrid SSD/HDD HPC storage system based on file size distributions2013
- A study on data deduplication in HPC storage systems2012
- P-Dedupe: Exploiting Parallelism in Data Deduplication System2012
- Middleware support for many-task computingCluster Computing, 2010