Yair Cohen

Storm-scale convection-allowing models (CAMs) explicitly resolve convective dynamics within the atmosphere to predict the evolution of thunderstorms and mesoscale convective systems that result in damaging extreme weather. Deep learning models have, thus far, not proven skillful in this regime of kilometer-scale atmospheric simulation, despite being competitive at coarser resolutions with state-of-the-art global, medium-range weather forecasting. We present a generative diffusion model called StormCast, which emulates the High-Resolution Rapid Refresh (HRRR)—National Oceanic and Atmospheric Administration’s state-of-the-art 3-kilometer operational CAM. StormCast autoregressively predicts 99 state variables at the kilometer scale using a 1-hour time step, with dense vertical resolution in the atmospheric boundary layer, conditioned on 26 synoptic variables. We show successfully learned kilometer-scale dynamics including competitive 1- to 6-hour forecast skill for composite radar reflectivity alongside physically realistic convective cluster evolution, moist updrafts, and cold pool morphology. These results present opportunities for improving kilometer-scale regional ML weather prediction and future climate hazard dynamical downscaling.

Data assimilation of observations into full atmospheric states is essential for weather forecast model initialization. Recently, methods for deep generative data assimilation have been proposed which allow for using new input data without retraining the model. They could also dramatically accelerate the costly data assimilation process used in operational regional weather models. Here, in a central US testbed, we demonstrate the viability of Score‐based Data Assimilation (SDA) in the context of realistically complex km‐scale weather. We train an unconditional diffusion model to generate snapshots of a state‐of‐the‐art km‐scale analysis product, the High Resolution Rapid Refresh. Then, using SDA to incorporate sparse weather station data, the model produces maps of precipitation and surface winds. The generated fields display physically plausible structures, such as gust fronts, and sensitivity tests confirm learned physics through multivariate relationships. Preliminary skill analysis shows the approach already outperforms a naive baseline of the High‐Resolution Rapid Refresh system itself. By incorporating observations from 40 weather stations, 10% lower RMSEs on left‐out stations are attained. Despite some lingering imperfections such as insufficiently disperse ensemble DA estimates, we find the results overall an encouraging proof of concept, and the first at km‐scale. It is a ripe time to explore extensions that combine increasingly ambitious regional state generators with an increasing set of in situ, ground‐based, and satellite remote sensing data streams.

This work integrates machine learning into an atmospheric parameterization to target uncertain mixing processes while maintaining interpretable, predictive, and well‐established physical equations. We adopt an eddy‐diffusivity mass‐flux (EDMF) parameterization for the unified modeling of various convective and turbulent regimes. To avoid drift and instability that plague offline‐trained machine learning parameterizations that are subsequently coupled with climate models, we frame learning as an inverse problem: Data‐driven models are embedded within the EDMF parameterization and trained online in a one‐dimensional vertical global climate model (GCM) column. Training is performed against output from large‐eddy simulations (LES) forced with GCM‐simulated large‐scale conditions in the Pacific. Rather than optimizing subgrid‐scale tendencies, our framework directly targets climate variables of interest, such as the vertical profiles of entropy and liquid water path. Specifically, we use ensemble Kalman inversion to simultaneously calibrate both the EDMF parameters and the parameters governing data‐driven lateral mixing rates. The calibrated parameterization outperforms existing EDMF schemes, particularly in tropical and subtropical locations of the present climate, and maintains high fidelity in simulating shallow cumulus and stratocumulus regimes under increased sea surface temperatures from AMIP4K experiments. The results showcase the advantage of physically constraining data‐driven models and directly targeting relevant variables through online learning to build robust and stable machine learning parameterizations.

Most machine learning applications in Earth system modeling currently rely on gradient‐based supervised learning. This imposes stringent constraints on the nature of the data used for training (typically, residual time tendencies are needed), and it complicates learning about the interactions between machine‐learned parameterizations and other components of an Earth system model. Approaching learning about process‐based parameterizations as an inverse problem resolves many of these issues, since it allows parameterizations to be trained with partial observations or statistics that directly relate to quantities of interest in long‐term climate projections. Here, we demonstrate the effectiveness of Kalman inversion methods in treating learning about parameterizations as an inverse problem. We consider two different algorithms: unscented and ensemble Kalman inversion. Both methods involve highly parallelizable forward model evaluations, converge exponentially fast, and do not require gradient computations. In addition, unscented Kalman inversion provides a measure of parameter uncertainty. We illustrate how training parameterizations can be posed as a regularized inverse problem and solved by ensemble Kalman methods through the calibration of an eddy‐diffusivity mass‐flux scheme for subgrid‐scale turbulence and convection, using data generated by large‐eddy simulations. We find the algorithms amenable to batching strategies, robust to noise and model failures, and efficient in the calibration of hybrid parameterizations that can include empirical closures and neural networks.

Most machine learning applications in Earth system modeling currently rely on gradient-based supervised learning. This imposes stringent constraints on the nature of the data used for training (typically, residual time tendencies are needed), and it complicates learning about the interactions between machine-learned parameterizations and other components of an Earth system model. Approaching learning about process-based parameterizations as an inverse problem resolves many of these issues, since it allows parameterizations to be trained with partial observations or statistics that directly relate to quantities of interest in long-term climate projections. Here we demonstrate the effectiveness of Kalman inversion methods in treating learning about parameterizations as an inverse problem. We consider two different algorithms: unscented and ensemble Kalman inversion. Both methods involve highly parallelizable forward model evaluations, converge exponentially fast, and do not require gradient computations. In addition, unscented Kalman inversion provides a measure of parameter uncertainty. We illustrate how training parameterizations can be posed as a regularized inverse problem and solved by ensemble Kalman methods through the calibration of an eddy-diffusivity mass-flux scheme for subgrid-scale turbulence and convection, using data generated by large-eddy simulations. We find the algorithms amenable to batching strategies, robust to noise and model failures, and efficient in the calibration of hybrid parameterizations that can include empirical closures and neural networks.

Most machine learning applications in Earth system modeling currently rely on gradient-based supervised learning. This imposes stringent constraints on the nature of the data used for training (typically, residual time tendencies are needed), and it complicates learning about the interactions between machine-learned parameterizations and other components of an Earth system model. Approaching learning about process-based parameterizations as an inverse problem resolves many of these issues, since it allows parameterizations to be trained with partial observations or statistics that directly relate to quantities of interest in long-term climate projections. Here we demonstrate the effectiveness of Kalman inversion methods in treating learning about parameterizations as an inverse problem. We consider two different algorithms: unscented and ensemble Kalman inversion. Both methods involve highly parallelizable forward model evaluations, converge exponentially fast, and do not require gradient computations. In addition, unscented Kalman inversion provides a measure of parameter uncertainty. How learning about parameterizations can be posed as an inverse problem and solved by ensemble Kalman methods is illustrated through the calibration of an eddy-diffusivity mass-flux scheme for subgrid-scale turbulence and convection, using data generated by large-eddy simulations. We find the algorithms amenable to batching strategies, robust to noise and model failures, and efficient in the calibration of hybrid parameterizations that can include empirical closures and neural networks.

Convection parameterizations such as eddy-diffusivity mass-flux (EDMF) schemes require a consistent closure formulation for the perturbation pressure, which arises in the equations for vertical momentum and turbulence kinetic energy (TKE). Here we derive an expression for the perturbation pressure from approximate analytical solutions for 2D and 3D thermal bubbles. The new closure combines modified pressure drag and virtual mass effects with a new momentum advection damping term. This advection damping is an important source in the lower half of the thermal bubble and at cloud base levels in convective systems. It represents the effect of the perturbation pressure to ensure the non-divergent properties of the flow. The new formulation represents the pressure drag to be inversely proportional to updraft depth. This is found to significantly improve simulations of the diurnal cycle of deep convection, without compromising simulations of shallow convection. It is thus a key step toward a unified scheme for a range of convective motions. By assuming that the pressure only redistributes TKE between updrafts and the environment laterally, a closure for the velocity pressure-gradient correlation is obtained from the perturbation pressure closure. This novel pressure closure is implemented in an extended EDMF scheme and is shown to successfully simulate a rising bubble as well as shallow and deep convection in a single column model.

Convection parameterizations such as eddy-diffusivity mass-flux (EDMF) schemes require a consistent closure formulation for the perturbation pressure, which arises in the equations for vertical momentum and turbulence kinetic energy (TKE). Here we derive an expression for the perturbation pressure from approximate analytical solutions for 2D and 3D rising thermal bubbles. The new closure combines a modified pressure drag and virtual mass effects with a new momentum advection term. This momentum advection is an important source in the lower half of the thermal bubble and at cloud base levels in convective systems. It represents the essential physics of the perturbation pressure, that is, to ensure the 3D non-divergent properties of the flow. Moreover, the new formulation modifies the pressure drag to be inversely proportional to updraft depth. This is found to significantly improve simulations of the diurnal cycle of deep convection, without compromising simulations of shallow convection. It is thus a key step toward a unified scheme for a range of convective motions. By assuming that the pressure only redistributes TKE between plumes and the environment, rather than vertically, a closure for the velocity pressure-gradient correlation is obtained from the perturbation pressure closure. This novel pressure closure is implemented in an extended EDMF scheme and is shown to successfully simulate a rising bubble test case as well as shallow and deep convection cases in a single column model.

Because of their limited spatial resolution, numerical weather prediction and climate models have to rely on parameterizations to represent atmospheric turbulence and convection. Historically, largely independent approaches have been used to represent boundary layer turbulence and convection, neglecting important interactions at the subgrid scale. Here we build on an eddy‐diffusivity mass‐flux (EDMF) scheme that represents all subgrid‐scale mixing in a unified manner, partitioning subgrid‐scale fluctuations into contributions from local diffusive mixing and coherent advective structures and allowing them to interact within a single framework. The EDMF scheme requires closures for the interaction between the turbulent environment and the plumes and for local mixing. A second‐order equation for turbulence kinetic energy (TKE) provides one ingredient for the diffusive local mixing closure, leaving a mixing length to be parameterized. Here, we propose a new mixing length formulation, based on constraints derived from the TKE balance. It expresses local mixing in terms of the same physical processes in all regimes of boundary layer flow. The formulation is tested at a range of resolutions and across a wide range of boundary layer regimes, including a stably stratified boundary layer, a stratocumulus‐topped marine boundary layer, and dry convection. Comparison with large eddy simulations (LES) shows that the EDMF scheme with this diffusive mixing parameterization accurately captures the structure of the boundary layer and clouds in all cases considered.