Abstract
As the resolution of weather and climate simulations increases, the amount of data produced is growing rapidly from hundreds of terabytes to tens of petabytes. The huge size becomes a limiting factor for broader adoption, and its fast growth rate will soon exhaust all available storage devices. To address these issues, we present EBCC (Error Bounded Climate-data Compressor). It follows a two-layer compression approach: a base compression layer using JPEG2000 to capture the bulk of the data with a high compression ratio, and a residual compression layer using wavelet transform and SPIHT (Set Partitioning In Hierarchical Trees) encoding to efficiently eliminate long-tail extreme errors. EBCC outperforms other methods in the benchmarks at relative error targets ranging from 0.1% to 10%. In the energy budget closure and Lagrangian trajectory benchmarks, it can achieve more than 100× compression while keeping errors within the natural variability derived from ERA5 uncertainty members. We implement EBCC as a standalone C library which is seamlessly integrated with NetCDF and Zarr pipelines.