H5Intent: Autotuning HDF5 With User Intent
- 6 November 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 36 (2) , 108-119
Abstract
The complexity of data management in HPC systems stems from the diversity in I/O behavior exhibited by new workloads, multistage workflows, and multitiered storage systems. The HDF5 library is a popular interface to interact with storage systems in HPC workloads. The library manages the complexity of diverse I/O behaviors by providing user-level configurations to optimize the I/O for HPC workloads. The HDF5 library exposes hundreds of configuration properties that can be set to alter how HDF5 manages I/O requests for better performance. However, determining which properties to set is quite challenging for users who lack expertise in HDF5 library internals. We propose a paradigm change through our H5Intent software, where users specify the intent of I/O operations and the software can set various HDF5 properties automatically to optimize the I/O behavior. This work demonstrates several use cases where mapping user-defined intents to HDF5 properties can be exploited to optimize I/O. In this study, we make three observations. First, I/O intents can accurately define HDF5 properties while managing conflicts between various properties and improving the I/O performance of microbenchmarks by up to 22×. Second, I/O intents can be efficiently passed to HDF5 with a small footprint of 6.74MB per node for thousands of intents per process. Third, an H5Intent VOL connector can dynamically map I/O intents to HDF5 properties for various I/O behaviors exhibited by our microbenchmark and improve I/O performance by up to 8.8×. Overall, H5Intent software improves the I/O performance of complex large-scale workloads we studied by up to 11×.Keywords
Funding Information
- U.S. Department of Energy
- Lawrence Livermore National Laboratory (DE-AC52-07NA27344)
- U.S. Department of Energy
- Office of Science
- Office of Advanced Scientific Computing Research
- DOE Early Career research Program (LLNL-CONF-849011)
This publication has 21 references indexed in Scilit:
- Mimir: Extending I/O Interfaces to Express User Intent for Complex Workloads in HPC2023
- UnifyFS: A User-level Shared File System for Unified Access to Distributed Local Storage2023
- Extracting and characterizing I/O behavior of HPC workloads2022
- A Comparative Study of MPI-IO Libraries for Offloading of Collective I/O Tasks2021
- Kill Two Birds with One Stone: Auto-tuning RocksDB for High Bandwidth and Low Latency2020
- Recorder 2.0: Efficient Parallel I/O Tracing and Analysis2020
- HCompress: Hierarchical Data Compression for Multi-Tiered Storage Environments2020
- Big data and HPC collocation: Using HPC idle resources for Big Data analytics2017
- A multi-level approach for understanding I/O activity in HPC applications2013
- 24/7 Characterization of petascale I/O workloads2009