Runway: In-transit Data Compression on Heterogeneous HPC Systems
Authors:
- 1 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 229-239
Abstract
To alleviate bottlenecks in storing and accessing data on high-performance computing (HPC) systems, I/O libraries are enabling computation while data is in-transit, such as HDFS filters. For scientific applications that commonly use floating-point data, error-bounded lossy compression methods are a critical technique to significantly reduce the storage and bandwidth requirements. Thus far, deciding when and where to schedule in-transit data transformations, such as compression, has been outside the scope of I/O libraries. In this paper, we introduce Runway, a runtime framework that enables computation on in-transit data with an object storage abstraction. Runway is designed to be extensible to execute user-defined functions at runtime. In this effort, we focus on studying methods to offload data compression operations to available processing units based on latency and throughput. We compare the performance of running compression on multi-core CPUs, as well as offloading it to a GPU and a Data Processing Unit (DPU). We implement a state-of-the-art error-bounded lossy compression algorithm, SZ3, as a Runway function with a variant optimized for DPUs. We propose dynamic modeling to guide scheduling decisions for in-transit data compression. We evaluate Runway using four scientific datasets from the SDRBench benchmark suite on a the Perlmutter supercomputer at NERSC.Keywords
Funding Information
- North Carolina State University (CNS-1812727)
- National Science Foundation (CNS-1812727)
- Ohio State University (GR130303)
- U.S. Department of Energy (DOE) (DE-AC02-05CH11231)
- Office of Science (DE-AC02-05CH11231)
- Office of Advanced Scientific Computing Research (ASCR) (DE-AC02-05CH11231)
This publication has 16 references indexed in Scilit:
- AMReX: Block-structured adaptive mesh refinement for multiphysics applicationsThe International Journal of High Performance Computing Applications, 2021
- SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors2020
- GPU Direct I/O with HDF52020
- cuSZ2020
- Parallel Query Service for Object-centric Data Management Systems2020
- Error Analysis of ZFP Compression for Floating-Point DataSIAM Journal on Scientific Computing, 2019
- Toward Scalable and Asynchronous Object-Centric Data Management for HPC2018
- Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworksConcurrency and Computation: Practice and Experience, 2013
- An overview of the HDF5 technology suite and its applications2011
- The BBP Algorithm for Pi2006