Allison H. Baker
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Allison H. Baker has a total of 172 co-authors. Here are the top co-authors by number of shared publications:
- Dorit Hammerling
Department of Applied Mathematics and Statistics, Colorado School of Mines, Golden, USA
16 shared publications - Bryan N. Lawrence
National Centre for Atmospheric Science, Department of Meteorology, University of Reading, Reading, UK
6 shared publications Guangwen Yang
International Laboratory for High-Resolution Earth System Model and Prediction (iHESP), Qingdao, China
6 shared publicationsHaiying Xu
6 shared publications- Juniper Tyree
INAR, University of Helsinki, Helsinki, Finland
6 shared publications - Milan Klöwer
University of Oxford, Oxford, United Kingdom
6 shared publications
Latest Publications
- Open Access
- 12 November 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-16
Comment on egusphere-2026-6708
Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.
- 12 November 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-16
Open AccessComment on egusphere-2026-6708
Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.
- Open Access
- 10 November 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-16
Comment on egusphere-2026-6708
Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.
- 10 November 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-16
Open AccessComment on egusphere-2026-6708
Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.
- Open Access
- 5 November 2026
- Published by Copernicus GmbH in EGUsphere
- Vol. 2026, 1-16
How Much Lossy Compression of Training Data Can An Atmosphere Emulator Tolerate?
Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.
- 5 November 2026
- Published by Copernicus GmbH in EGUsphere
- Vol. 2026, 1-16
Open AccessHow Much Lossy Compression of Training Data Can An Atmosphere Emulator Tolerate?
Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.
- Open Access
- 7 July 2026
- Published by Copernicus GmbH in Geoscientific Model Development
- Vol. 19 (13) , 5933-5960
ClimateBenchPress (v1.0): a benchmark for lossy compression of climate data
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 7 July 2026
- Published by Copernicus GmbH in Geoscientific Model Development
- Vol. 19 (13) , 5933-5960
Open AccessClimateBenchPress (v1.0): a benchmark for lossy compression of climate data
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Reply on RC1
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Open AccessReply on RC1
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Reply on RC2
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Open AccessReply on RC2
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 19 April 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Comment on egusphere-2026-60
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 19 April 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Open AccessComment on egusphere-2026-60
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 13 March 2026
- Published by Copernicus GmbH
Application of advanced lossy compression in the NetCDF ecosystem for CONUS404 data
Many geoscientific datasets, such as those produced by climate and weather models, are stored in the NetCDF file format. These datasets are typically very large and often strain institutional data storage resources. While lossy compression methods for scientific data have become more studied and adopted in recent years, most advanced lossy approaches do not work easily and/or transparently with NetCDF files. For example, they may require a file format conversion or they may not work correctly with “missing values” or “fill values” that are often present in model outputs. While lossy quantization approaches such at BitRound and Granular BitRound have built-in support by NetCDF and are quite easy to use, such approaches are generally not able to reduce the data size as much as more advanced compressors (for a fixed error metric), like SPERR, ZFP, or SZ3.We are particularly interested in reducing the data size of the CONUS404 dataset. CONUS404 is a publicly available unique high-resolution hydro-climate dataset produced by Weather Research and Forecasting (WRF) Model simulations that cover the CONtiguous United States (CONUS) for 40 years at 4-km resolution (a collaboration between NSF National Center for Atmospheric Research the U.S. Geological Survey Water Mission Area). Here, we investigate one advanced lossy compressor, SPERR [1], together with its plugin for NetCDF files, H5Z-SPERR [2], in a Python-based workflow to compress and analyze CONUS404 data. SPERR is attractive due to its support for quality control in terms of both maximum point-wise error (PWE) and peak signal-to-noise ratio (PSNR), enabling easy experimenting of storage-quality tradeoffs. Further, given a target quality metric, previous work has shown that SPERR likely produces the smallest compressed file size compared to other advanced compressors. It leverages the HDF5 dynamic plugin mechanism to enable users to stay in the NetCDF ecosystem with minimal to no change to existing analysis workflows, whenever a typical NetCDF file is able to be read. And, importantly for our work, the SPERR plugin supports efficient masking of “missing values,” which are common to climate and weather model output. The support for missing values enables compression on many variables which are not naturally handled by other advanced compressors that rely on HDF5 plugins. Further, because H5Z-SPERR directly handles missing values, they can be stored in a much more compact format (and are restored during decompression), further improving compression efficiency. (Note that built-in NetCDF quantization approaches can work with missing values.) Our experimentation demonstrates the benefit of enabling advanced lossy (de)compression in the NetCDF ecosystem: adoption friction is kept at the minimum with little change to workflows, while storage requirements are greatly reduced. [1] https://github.com/NCAR/SPERR[2] https://github.com/NCAR/H5Z-SPERR
- 13 March 2026
- Published by Copernicus GmbH
Open AccessApplication of advanced lossy compression in the NetCDF ecosystem for CONUS404 data
Many geoscientific datasets, such as those produced by climate and weather models, are stored in the NetCDF file format. These datasets are typically very large and often strain institutional data storage resources. While lossy compression methods for scientific data have become more studied and adopted in recent years, most advanced lossy approaches do not work easily and/or transparently with NetCDF files. For example, they may require a file format conversion or they may not work correctly with “missing values” or “fill values” that are often present in model outputs. While lossy quantization approaches such at BitRound and Granular BitRound have built-in support by NetCDF and are quite easy to use, such approaches are generally not able to reduce the data size as much as more advanced compressors (for a fixed error metric), like SPERR, ZFP, or SZ3.We are particularly interested in reducing the data size of the CONUS404 dataset. CONUS404 is a publicly available unique high-resolution hydro-climate dataset produced by Weather Research and Forecasting (WRF) Model simulations that cover the CONtiguous United States (CONUS) for 40 years at 4-km resolution (a collaboration between NSF National Center for Atmospheric Research the U.S. Geological Survey Water Mission Area). Here, we investigate one advanced lossy compressor, SPERR [1], together with its plugin for NetCDF files, H5Z-SPERR [2], in a Python-based workflow to compress and analyze CONUS404 data. SPERR is attractive due to its support for quality control in terms of both maximum point-wise error (PWE) and peak signal-to-noise ratio (PSNR), enabling easy experimenting of storage-quality tradeoffs. Further, given a target quality metric, previous work has shown that SPERR likely produces the smallest compressed file size compared to other advanced compressors. It leverages the HDF5 dynamic plugin mechanism to enable users to stay in the NetCDF ecosystem with minimal to no change to existing analysis workflows, whenever a typical NetCDF file is able to be read. And, importantly for our work, the SPERR plugin supports efficient masking of “missing values,” which are common to climate and weather model output. The support for missing values enables compression on many variables which are not naturally handled by other advanced compressors that rely on HDF5 plugins. Further, because H5Z-SPERR directly handles missing values, they can be stored in a much more compact format (and are restored during decompression), further improving compression efficiency. (Note that built-in NetCDF quantization approaches can work with missing values.) Our experimentation demonstrates the benefit of enabling advanced lossy (de)compression in the NetCDF ecosystem: adoption friction is kept at the minimum with little change to workflows, while storage requirements are greatly reduced. [1] https://github.com/NCAR/SPERR[2] https://github.com/NCAR/H5Z-SPERR
- Open Access
- 3 March 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Comment on egusphere-2026-60
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 3 March 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Open AccessComment on egusphere-2026-60
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 9 February 2026
- Published by Copernicus GmbH in EGUsphere
- Vol. 2026, 1-34
ClimateBenchPress (v1.0): A Benchmark for Lossy Compression of Climate Data
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 9 February 2026
- Published by Copernicus GmbH in EGUsphere
- Vol. 2026, 1-34
Open AccessClimateBenchPress (v1.0): A Benchmark for Lossy Compression of Climate Data
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.