ABAllison H. Baker

Allison H. Baker

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Allison H. Baker has a total of 172 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Comment on egusphere-2026-6708

    by Finn Rebassoo, Peter Caldwell, Peter Lindstrom, Allison Baker, Elynn Wu

    Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.

  • Open Access

    Comment on egusphere-2026-6708

    by Finn Rebassoo, Peter Caldwell, Peter Lindstrom, Allison Baker, Elynn Wu

    Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.

  • Open Access

    How Much Lossy Compression of Training Data Can An Atmosphere Emulator Tolerate?

    by Finn Rebassoo, Peter Caldwell, Peter Lindstrom, Allison Baker, Elynn Wu

    Training machine learning (ML) emulators on km-scale Earth system models (ESMs) would require more data than is practical to store. Lossy data compression is a natural solution to this problem, but adoption by the ESM community has been slow, partly because of the fear of negatively impacting scientific conclusions. In this study, we demonstrate what level of lossy compression is acceptable for an ML-based emulator of the E3SMv3 atmospheric model at 1° resolution. We specifically use the ZFP compressor, which provides state-of-the-art performance and is integrated into NetCDF4. Thus, ZFP can be applied to the E3SM data output directly and the compressed data read easily with Python tools. We measure lossy compression impact by looking at the skill of the ACE2-emulator for AMIP-like runs. We find that a compression rate of roughly 5x does not affect the physics predictions of the ACE2-emulator. As we compress the data more aggressively, some metrics start to degrade, and at a compression rate of 11x many metrics have become unacceptable.

  • Open Access

    ClimateBenchPress (v1.0): a benchmark for lossy compression of climate data

    by Tim Reichelt, Juniper Tyree, Milan Klöwer, Peter Dueben, Bryan N. Lawrence, Allison H. Baker, Sara Faghih-Naini, Torsten Hoefler, Philip Stier

    The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.


  • Open Access

    Reply on RC1

    by Tim Reichelt, Juniper Tyree, Milan Klöwer, Peter Dueben, Bryan N. Lawrence, Allison H. Baker, Sara Faghih-Naini, Torsten Hoefler, Philip Stier

    The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.

  • Open Access

    Reply on RC2

    by Tim Reichelt, Juniper Tyree, Milan Klöwer, Peter Dueben, Bryan N. Lawrence, Allison H. Baker, Sara Faghih-Naini, Torsten Hoefler, Philip Stier

    The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.

  • Open Access

    Comment on egusphere-2026-60

    by Tim Reichelt, Juniper Tyree, Milan Klöwer, Peter Dueben, Bryan N. Lawrence, Allison H. Baker, Sara Faghih-Naini, Torsten Hoefler, Philip Stier

    The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.

  • Open Access

    Application of advanced lossy compression in the NetCDF ecosystem for CONUS404 data

    by Shaomeng Li, Allison Baker, Lulin Xue

    Many geoscientific datasets, such as those produced by climate and weather models, are stored in the NetCDF file format. These datasets are typically very large and often strain institutional data storage resources. While lossy compression methods for scientific data have become more studied and adopted in recent years, most advanced lossy approaches do not work easily and/or transparently with NetCDF files. For example, they may require a file format conversion or they may not work correctly with “missing values” or “fill values” that are often present in model outputs. While lossy quantization approaches such at BitRound and Granular BitRound have built-in support by NetCDF and are quite easy to use, such approaches are generally not able to reduce the data size as much as more advanced compressors (for a fixed error metric), like SPERR, ZFP, or SZ3.We are particularly interested in reducing the data size of the CONUS404 dataset. CONUS404 is a publicly available unique high-resolution hydro-climate dataset produced by Weather Research and Forecasting (WRF) Model simulations that cover the CONtiguous United States (CONUS) for 40 years at 4-km resolution (a collaboration between NSF National Center for Atmospheric Research the U.S. Geological Survey Water Mission Area). Here, we investigate one advanced lossy compressor, SPERR [1], together with its plugin for NetCDF files, H5Z-SPERR [2], in a Python-based workflow to compress and analyze CONUS404 data. SPERR is attractive due to its support for quality control in terms of both maximum point-wise error (PWE) and peak signal-to-noise ratio (PSNR), enabling easy experimenting of storage-quality tradeoffs. Further, given a target quality metric, previous work has shown that SPERR likely produces the smallest compressed file size compared to other advanced compressors. It leverages the HDF5 dynamic plugin mechanism to enable users to stay in the NetCDF ecosystem with minimal to no change to existing analysis workflows, whenever a typical NetCDF file is able to be read. And, importantly for our work, the SPERR plugin supports efficient masking of “missing values,” which are common to climate and weather model output. The support for missing values enables compression on many variables which are not naturally handled by other advanced compressors that rely on HDF5 plugins. Further, because H5Z-SPERR directly handles missing values, they can be stored in a much more compact format (and are restored during decompression), further improving compression efficiency. (Note that built-in NetCDF quantization approaches can work with missing values.) Our experimentation demonstrates the benefit of enabling advanced lossy (de)compression in the NetCDF ecosystem: adoption friction is kept at the minimum with little change to workflows, while storage requirements are greatly reduced. [1] https://github.com/NCAR/SPERR[2] https://github.com/NCAR/H5Z-SPERR

  • Open Access

    Comment on egusphere-2026-60

    by Tim Reichelt, Juniper Tyree, Milan Klöwer, Peter Dueben, Bryan N. Lawrence, Allison H. Baker, Sara Faghih-Naini, Torsten Hoefler, Philip Stier

    The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.

  • Open Access

    ClimateBenchPress (v1.0): A Benchmark for Lossy Compression of Climate Data

    by Tim Reichelt, Juniper Tyree, Milan Klöwer, Peter Dueben, Bryan N. Lawrence, Allison H. Baker, Sara Faghih-Naini, Torsten Hoefler, Philip Stier

    The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.