Milan Klöwer
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Milan Klöwer has a total of 258 co-authors. Here are the top co-authors by number of shared publications:
- Peter Dueben
ECMWF Reading UK
16 shared publications - Juniper Tyree
INAR, University of Helsinki, Helsinki, Finland
14 shared publications - Sara Faghih-Naini
European Centre for Medium-Range Weather Forecasts, Reading, United Kingdom
12 shared publications - Tim Reichelt
University of Oxford, Oxford, United Kingdom
12 shared publications - Bryan N. Lawrence
National Centre for Atmospheric Science, Department of Meteorology, University of Reading, Reading, UK
10 shared publications - Philip Stier
Department of Physics, University of Oxford, Oxford, UK
9 shared publications
Latest Publications
- Open Access
- 12 November 2026
- Published by Wiley
An online-learned eddy parameterization in a differentiable shallow water model
Ocean general circulation models rely on parameterizations to approximate subgrid-scale processes. While parameterizations are traditionally physics-based, machine learning has recently provided an alternative, data-driven approach for developing parameterizations. However, most machine learning-based parameterizations are learned offline due to the lack of differentiable general circulation models. Here, we train an eddy momentum parameterization represented by a neural network online in an idealized barotropic gyre model. We develop the model as fully differentiable in Julia with the automatic differentiation tool Enzyme. The network weights are trained using a loss function that has the same form as in adjoint-based data assimilation, with synthetic observations generated by a fine-resolution simulation. A range of diagnostics evaluate the performance of the parameterized coarse-resolution model. We find improved energetics, physically realistic dynamics, and numerical stability for multi-decadal roll-outs. Importantly, the online-learned parameterization absorbs artifacts from the numerical discretization, such as the time-stepping scheme used, which is invisible to offline-learned parameterizations. We suggest that this is key to improved numerical stability of machine-learned parameterizations. Our research suggests that online-trained neural network parameterizations can be an effective solution to the subgrid-scale parameterization and model calibration problem.
- 12 November 2026
- Published by Wiley
Open AccessAn online-learned eddy parameterization in a differentiable shallow water model
Ocean general circulation models rely on parameterizations to approximate subgrid-scale processes. While parameterizations are traditionally physics-based, machine learning has recently provided an alternative, data-driven approach for developing parameterizations. However, most machine learning-based parameterizations are learned offline due to the lack of differentiable general circulation models. Here, we train an eddy momentum parameterization represented by a neural network online in an idealized barotropic gyre model. We develop the model as fully differentiable in Julia with the automatic differentiation tool Enzyme. The network weights are trained using a loss function that has the same form as in adjoint-based data assimilation, with synthetic observations generated by a fine-resolution simulation. A range of diagnostics evaluate the performance of the parameterized coarse-resolution model. We find improved energetics, physically realistic dynamics, and numerical stability for multi-decadal roll-outs. Importantly, the online-learned parameterization absorbs artifacts from the numerical discretization, such as the time-stepping scheme used, which is invisible to offline-learned parameterizations. We suggest that this is key to improved numerical stability of machine-learned parameterizations. Our research suggests that online-trained neural network parameterizations can be an effective solution to the subgrid-scale parameterization and model calibration problem.
- Open Access
- 29 October 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-110
Reply on RC2
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- 29 October 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-110
Open AccessReply on RC2
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- Open Access
- 29 October 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-110
Reply on RC1
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- 29 October 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-110
Open AccessReply on RC1
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- Open Access
- 2 September 2026
- Published by Copernicus GmbH
- p. 1-110
Comment on egusphere-2026-4266
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- 2 September 2026
- Published by Copernicus GmbH
- p. 1-110
Open AccessComment on egusphere-2026-4266
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- Open Access
- 26 August 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-110
Comment on egusphere-2026-4266
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- 26 August 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-110
Open AccessComment on egusphere-2026-4266
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- Open Access
- 28 July 2026
- Published by Copernicus GmbH in EGUsphere
- Vol. 2026, 1-110
Compression Safeguards: Building Trust into Lossy Data Compression
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- 28 July 2026
- Published by Copernicus GmbH in EGUsphere
- Vol. 2026, 1-110
Open AccessCompression Safeguards: Building Trust into Lossy Data Compression
The growth in data volumes produced by scientific models is accelerating. Data production of high-resolution weather and climate models is outpacing the methods and budgets for storing, sharing, and analysing this data, posing a threat to scientific progress. Lossy data compression greatly reduces data sizes but loses some quality, detail, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the risk of losing important information, has thus far limited their adoption. We introduce Compression Safeguards, a user-centric and domain-independent framework to overcome this trust gap, with which (i) scientists declare their precise safety requirements for what lossy compression must preserve, e.g. regionally varying error bounds on quantities derived from the decompressed data, then (ii) wrap a compressor of their choice in the corresponding safeguards, which then (iii) guarantee that the safety requirements are always met by the safeguarded compressor, at most at the cost of a reduced compression ratio. The Compression Safeguards represent a paradigm shift: Data producers and users no longer carry the risks of lossy compression, having to manually check for problems after compression, but instead control up-front what needs to be safeguarded. With the appropriate safeguards, trust can grow in all lossy compressors and even untrusted, potentially unsafe compressors can be used safely and with confidence. Users thus no longer need to re-verify each new compressor for each new use case, or to restrict themselves to few safe compressors and supported use cases. We showcase how our reference implementation of the Compression Safeguards can be flexibly applied to safeguard important properties across several real-world examples from weather and climate sciences for different compressors. The impact on compression ratio varies but is small in many cases. The computational load increases during compression but is negligible during decompression. Altogether, Compression Safeguards provide a key modular tool that gives users the confidence to use lossy compression safely across scientific disciplines. Safeguards can unlock the data reduction benefits of lossy compression and therefore solve many data storage problems across the scientific community that otherwise hinder research.
- Open Access
- 7 July 2026
- Published by Copernicus GmbH in Geoscientific Model Development
- Vol. 19 (13) , 5933-5960
ClimateBenchPress (v1.0): a benchmark for lossy compression of climate data
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 7 July 2026
- Published by Copernicus GmbH in Geoscientific Model Development
- Vol. 19 (13) , 5933-5960
Open AccessClimateBenchPress (v1.0): a benchmark for lossy compression of climate data
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Reply on RC1
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Open AccessReply on RC1
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Reply on RC2
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- 13 June 2026
- Published by Copernicus GmbH
- Vol. 2026, 1-34
Open AccessReply on RC2
The rapidly growing volume of weather and climate data, both from models and observations, is increasing the pressure on data centers, restricting scientific analysis, and data distribution. For example, kilometre-scale climate models can generate petabytes of data per simulated month, making it generally infeasible to store all output. To address this challenge, numerous novel compression techniques have been proposed to ease data storage requirements. However, there exist no well-defined benchmarks for rigorously evaluating and comparing the performance of these compressors, including their impact on the data's properties. The lack of benchmarks makes it difficult to design and standardize compressors for weather and climate data, and for scientists to trust that compression errors have no significant impact on their analysis. Here, we address this gap by presenting ClimateBenchPress, a benchmark suite for lossy compression of climate data, which defines both data sets and evaluation techniques. The benchmark covers climate variables following various statistical distributions at medium to very high resolution in time and space, from both numerical models and satellite observations. To ensure a fair comparison between different compressors, each variable comes with a set of maximum error bound checks that the lossy compressors need to pass. By evaluating an initial set of baseline compressors on the benchmark, we gather practical insights for effective application of lossy compression. Our benchmark is open source and extensible: users can easily add new compressors, data sources, and evaluation metrics depending on their own specific use cases.
- Open Access
- 30 April 2026
- Published by Wiley
DJ4Earth: Differentiable, and Performance-portable Earth System Modeling via Program Transformations
Differentiable Earth system models (ESMs) enable powerful applications such as sensitivity analysis, gradient-based calibration, state estimation, boundary flux inversions, uncertainty quantification, and online machine learning. Reverse-mode automatic differentiation (AD) efficiently provides gradients for such tasks, yet models have rarely included this capability because of complex, bespoke numerical algorithms. As part of the \textit{Differentiable programming in Julia for Earth system modeling (DJ4Earth)} initiative, we present improved capabilities of the AD tool Enzyme.jl and the new compiler transpilation tool Reactant.jl, augmented by sophisticated checkpointing algorithms, which, together make general-purpose AD tractable and efficient for full-fledged ESM components written in Julia. Operating at the Low-Level Virtual Machine (LLVM) intermediate representation or Multi-Level Intermediate Representation (MLIR) compiler levels, these frameworks support mutable memory, custom kernels, and compiler optimizations before and after differentiation. Julia-specific challenges related to just-in-time compilation and garbage collection are handled efficiently. Reactant further enables automatic performance portability across CPUs, GPUs, and TPUs, facilitating use of emerging AI-customized high-performance computing architectures. We demonstrate these frameworks on four Julia-based ESM components featuring diverse spatial discretizations and numerical algorithms: (i) the rotating-sphere shallow water model ShallowWaters.jl, (ii) the finite-volume ocean model Oceananigans.jl, (iii) the finite-element ice sheet model DJUICE.jl, and (iv) the spectral atmospheric model SpeedyWeather.jl. Across these ESM components, our tools compute efficient and correct gradients. These results establish a foundation for differentiable, high-performance and performance-portable ESMs that can integrate neural networks for unresolved processes, trained online, enabling next-generation hybrid physics--machine learning ESMs constrained by physical dynamics and observations.
- 30 April 2026
- Published by Wiley
Open AccessDJ4Earth: Differentiable, and Performance-portable Earth System Modeling via Program Transformations
Differentiable Earth system models (ESMs) enable powerful applications such as sensitivity analysis, gradient-based calibration, state estimation, boundary flux inversions, uncertainty quantification, and online machine learning. Reverse-mode automatic differentiation (AD) efficiently provides gradients for such tasks, yet models have rarely included this capability because of complex, bespoke numerical algorithms. As part of the \textit{Differentiable programming in Julia for Earth system modeling (DJ4Earth)} initiative, we present improved capabilities of the AD tool Enzyme.jl and the new compiler transpilation tool Reactant.jl, augmented by sophisticated checkpointing algorithms, which, together make general-purpose AD tractable and efficient for full-fledged ESM components written in Julia. Operating at the Low-Level Virtual Machine (LLVM) intermediate representation or Multi-Level Intermediate Representation (MLIR) compiler levels, these frameworks support mutable memory, custom kernels, and compiler optimizations before and after differentiation. Julia-specific challenges related to just-in-time compilation and garbage collection are handled efficiently. Reactant further enables automatic performance portability across CPUs, GPUs, and TPUs, facilitating use of emerging AI-customized high-performance computing architectures. We demonstrate these frameworks on four Julia-based ESM components featuring diverse spatial discretizations and numerical algorithms: (i) the rotating-sphere shallow water model ShallowWaters.jl, (ii) the finite-volume ocean model Oceananigans.jl, (iii) the finite-element ice sheet model DJUICE.jl, and (iv) the spectral atmospheric model SpeedyWeather.jl. Across these ESM components, our tools compute efficient and correct gradients. These results establish a foundation for differentiable, high-performance and performance-portable ESMs that can integrate neural networks for unresolved processes, trained online, enabling next-generation hybrid physics--machine learning ESMs constrained by physical dynamics and observations.