Towards Green Scientific Data Compression Through High-Level I/O Interfaces
Authors:
- 1 September 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 209-216
Abstract
Every HPC system today has to cope with a deluge of data generated by scientific applications, simulations or large-scale experiments. The upscaling of supercomputer systems and infrastructures, generally results in a dramatic increase of their energy consumption. In this paper, we argue that techniques like data compression can lead to significant gains in terms of power efficiency by reducing both network and storage requirements. However, any data reduction is highly data specific and should comply with established requirements. Therefore, unsuitable or inappropriate compression strategy can utilize more resources and energy than necessary. To that end, we propose a novel methodology for achieving on-the-fly intelligent determination of energy efficient data reduction for a given data set by leveraging state-of-the-art compression algorithms and meta data at application-level I/O. We motivate our work by analyzing the energy and storage saving needs of data sets from real-life scientific HPC applications, and review the various lossless compression techniques that can be applied. We find that the resulting data reduction can decrease the data volume transferred and stored by as much as 80 % in some cases, consequently leading to significant savings in storage and networking costs.Keywords
This publication has 15 references indexed in Scilit:
- Design of energy-efficient discrete cosine transform using pruned arithmetic circuits2016
- ARDUPOWER: A low-cost wattmeter to improve energy efficiency of HPC applications2015
- A Best Practice Analysis of HDF$5$ and NetCDF-$4$ Using Lustre2015
- A Semantics-Aware I/O Interface for High Performance Computing2013
- Reducing the HPC-datastorage footprint with MAFISC—Multidimensional Adaptive Filtering Improved Scientific data CompressionComputer Science - Research and Development, 2012
- Parallel lossless data compression on the GPU2012
- Interannual variability of carbon fluxes in the North Sea from 1970 to 2006 – Competing effects of abiotic and biotic drivers on the gas-exchange of CO2Estuarine, Coastal and Shelf Science, 2012
- Improving I/O Forwarding Throughput with Data Compression2011
- Fast Lossless Compression of Scientific Floating-Point Data2006
- Hardware-assisted data compression for energy minimization in systems with embedded processors2002