Tobias Sebastian Finn

In supervised data assimilation machine learning emulation, the training data contain targets produced by an existing data assimilation scheme, such as analysis increments. By contrast, data assimilation networks were recently proposed to learn the analysis operator while they are embedded in the forecast–analysis cycle: their only targets are the true trajectory and the observations thereof. They are therefore trained to produce a stable and accurate sequential estimator, rather than to reproduce the output of a prescribed data assimilation algorithm. Conceptually more fundamental, yet computationally more challenging, such learned data assimilation scheme was shown to achieve accuracy comparable to that of the ensemble Kalman filter when applied to low-order chaotic dynamics. Strikingly, the same accuracy can be reached with a single state forecast instead of an ensemble, hence bypassing the need to explicitly represent forecast uncertainty. In this study, we extend the investigation of such learned analysis operators beyond the preliminary experiments reported so far. First, we analyse the emergence of local patterns encoded in the operator, which accounts for the remarkable scalability of the approach to high-dimensional state spaces. Second, we assess the performance of the learned operators in stronger nonlinear regimes of the chaotic dynamics. We show that they can match the efficiency of the iterative ensemble Kalman filter, the baseline in this context, while avoiding the need for nonlinear iterative optimisation. Throughout the paper, we seek underlying reasons for the efficiency of the approach, drawing on insights from both machine learning and nonlinear data assimilation.

Recent work has shown that the analysis operator in sequential data assimilation designed to track chaotic dynamics, can be learned with deep learning from the sole knowledge of a true state trajectory and observations thereof. This approach to learning the analysis is computationally more challenging, yet conceptually more fundamental than approaches that learn a direct mapping from forecasts and observations to the corresponding analysis increments. Such learned scheme has been demonstrated to achieve accuracy comparable to that of the ensemble Kalman filter when applied to low-order dynamics. Strikingly, the same accuracy can be reached with a single state forecast instead of an ensemble, hence bypassing the need to explicitly represent forecast uncertainty. In this study, we extend the investigation of such learned analysis operators beyond the preliminary experiments reported so far. First, we analyse the emergence of local patterns encoded in the operator, which accounts for the remarkable scalability of the approach to high-dimensional state spaces. Second, we assess the performance of the learned operators in stronger nonlinear regimes of the chaotic dynamics. We show that they can match the efficiency of the iterative ensemble Kalman filter, the baseline in this context, while avoiding the need for nonlinear iterative optimisation. Throughout the paper, we seek underlying reasons for the efficiency of the approach, drawing on insights from both machine learning and nonlinear data assimilation.

Recent work has shown that the analysis operator in sequential data assimilation designed to track chaotic dynamics, can be learned with deep learning from the sole knowledge of a true state trajectory and observations thereof. This approach to learning the analysis is computationally more challenging, yet conceptually more fundamental than approaches that learn a direct mapping from forecasts and observations to the corresponding analysis increments. Such learned scheme has been demonstrated to achieve accuracy comparable to that of the ensemble Kalman filter when applied to low-order dynamics. Strikingly, the same accuracy can be reached with a single state forecast instead of an ensemble, hence bypassing the need to explicitly represent forecast uncertainty. In this study, we extend the investigation of such learned analysis operators beyond the preliminary experiments reported so far. First, we analyse the emergence of local patterns encoded in the operator, which accounts for the remarkable scalability of the approach to high-dimensional state spaces. Second, we assess the performance of the learned operators in stronger nonlinear regimes of the chaotic dynamics. We show that they can match the efficiency of the iterative ensemble Kalman filter, the baseline in this context, while avoiding the need for nonlinear iterative optimisation. Throughout the paper, we seek underlying reasons for the efficiency of the approach, drawing on insights from both machine learning and nonlinear data assimilation.

Recent work has shown that the analysis operator in sequential data assimilation designed to track chaotic dynamics, can be learned with deep learning from the sole knowledge of a true state trajectory and observations thereof. This approach to learning the analysis is computationally more challenging, yet conceptually more fundamental than approaches that learn a direct mapping from forecasts and observations to the corresponding analysis increments. Such learned scheme has been demonstrated to achieve accuracy comparable to that of the ensemble Kalman filter when applied to low-order dynamics. Strikingly, the same accuracy can be reached with a single state forecast instead of an ensemble, hence bypassing the need to explicitly represent forecast uncertainty. In this study, we extend the investigation of such learned analysis operators beyond the preliminary experiments reported so far. First, we analyse the emergence of local patterns encoded in the operator, which accounts for the remarkable scalability of the approach to high-dimensional state spaces. Second, we assess the performance of the learned operators in stronger nonlinear regimes of the chaotic dynamics. We show that they can match the efficiency of the iterative ensemble Kalman filter, the baseline in this context, while avoiding the need for nonlinear iterative optimisation. Throughout the paper, we seek underlying reasons for the efficiency of the approach, drawing on insights from both machine learning and nonlinear data assimilation.

This study explores a physics–data‐driven hybrid approach for sea‐ice column physics models, in which a machine‐learning component acts as a state‐dependent parametrization of forecast errors. We examine how perturbations in snow thermodynamics and sea‐ice radiative properties affect forecast errors, and train dedicated neural networks (NNs) for each model configuration. The performance of the hybrid models is evaluated for long lead‐time forecasts and compared against a benchmark system based on climatological forecast‐error estimates. The NN‐based hybrids prove to be stable, robust to initial condition and atmospheric forcing errors, and consistently outperform their climatology‐based counterpart. To derive guiding principles for efficiently handling possible physical model updates, we perform transfer learning experiments to test whether pretrained NNs optimized for one model configuration can be successfully adapted to another. Results indicate that direct evaluation of pretrained networks on the target task provides useful insights into their adaptability, recommending transfer learning whenever performance exceeds a trivial baseline. Finally, a feature‐importance analysis shows that atmospheric forcing inputs have negligible influence on NN predictive skill, and ice‐layer enthalpies play a key role in achieving satisfactory performance.

Recent work has shown that the analysis operator in sequential data assimilation designed to track chaotic dynamics, can be learned with deep learning from the sole knowledge of a true state trajectory and observations thereof. This approach to learning the analysis is computationally more challenging, yet conceptually more fundamental than approaches that learn a direct mapping from forecasts and observations to the corresponding analysis increments. Such learned scheme has been demonstrated to achieve accuracy comparable to that of the ensemble Kalman filter when applied to low-order dynamics. Strikingly, the same accuracy can be reached with a single state forecast instead of an ensemble, hence bypassing the need to explicitly represent forecast uncertainty. In this study, we extend the investigation of such learned analysis operators beyond the preliminary experiments reported so far. First, we analyse the emergence of local patterns encoded in the operator, which accounts for the remarkable scalability of the approach to high-dimensional state spaces. Second, we assess the performance of the learned operators in stronger nonlinear regimes of the chaotic dynamics. We show that they can match the efficiency of the iterative ensemble Kalman filter, the baseline in this context, while avoiding the need for nonlinear iterative optimisation. Throughout the paper, we seek underlying reasons for the efficiency of the approach, drawing on insights from both machine learning and nonlinear data assimilation.

We propose a stochastic framework for wildfire spread prediction using deep generative diffusion models with ensemble sampling. In contrast to traditional deterministic approaches that struggle to capture the inherent uncertainty and variability of wildfire dynamics, our method generates probabilistic forecasts by sampling multiple plausible future scenarios conditioned on the same initial state. As a proof-of-concept, the model is trained on synthetic wildfire data generated by a probabilistic cellular automata simulator conditioned on canopy cover, vegetation density, and terrain slope for two real fires, namely the Chimney Fire in 2016 and the Ferguson Fire in 2018, both in California. To assess predictive performance and uncertainty representation under an identical neural network architecture, we compare a conventional supervised regression training paradigm against a conditional diffusion framework that employs ensemble sampling, and evaluate both approaches on independent ensemble test datasets. Across independent ensemble test sets, the diffusion surrogate consistently outperforms the deterministic baseline. It delivers lower errors in standard accuracy metrics such as mean squared error (MSE), exhibits higher spatial coherence as reflected by improved structural similarity index measure (SSIM) values, and generates samples of superior distributional quality according to the Fréchet inception distance (FID). Moreover, the diffusion-based model shows stronger probabilistic capability, as evidenced by higher scores in the hit rate (HR) metric, which we introduce as an uncertainty-aware verification measure. These results demonstrate that diffusion-based ensemble modelling provides a more flexible and effective approach for wildfire forecasting and, by capturing the distributional characteristics of future fire states, supports the generation of fire susceptibility maps that convey probabilistic risk information useful for assessment and operational planning in fire-prone environments.

Recent work has shown that the analysis operator in sequential data assimilation designed to track chaotic dynamics, can be learned with deep learning from the sole knowledge of a true state trajectory and observations thereof. This approach to learning the analysis is computationally more challenging, yet conceptually more fundamental than approaches that learn a direct mapping from forecasts and observations to the corresponding analysis increments. Such learned scheme has been demonstrated to achieve accuracy comparable to that of the ensemble Kalman filter when applied to low-order dynamics. Strikingly, the same accuracy can be reached with a single state forecast instead of an ensemble, hence bypassing the need to explicitly represent forecast uncertainty. In this study, we extend the investigation of such learned analysis operators beyond the preliminary experiments reported so far. First, we analyse the emergence of local patterns encoded in the operator, which accounts for the remarkable scalability of the approach to high-dimensional state spaces. Second, we assess the performance of the learned operators in stronger nonlinear regimes of the chaotic dynamics. We show that they can match the efficiency of the iterative ensemble Kalman filter, the baseline in this context, while avoiding the need for nonlinear iterative optimisation. Throughout the paper, we seek underlying reasons for the efficiency of the approach, drawing on insights from both machine learning and nonlinear data assimilation.

Popular (ensemble) Kalman filter data assimilation (DA) approaches assume that the errors in both the a priori estimate of the state and the observations are Gaussian. For constrained variables, for example, sea‐ice concentration or stress, such an assumption does not hold. The variational autoencoder (VAE) is a machine‐learning (ML) technique that allows us to map an arbitrary distribution to/from a latent space in which the distribution is supposedly closer to a Gaussian. We propose a novel hybrid DA–ML approach in which VAEs are incorporated in the DA procedure. Specifically, we introduce a variant of the popular ensemble transform Kalman filter (ETKF) in which the analysis is applied in the latent space of a single VAE or a pair of VAEs. In twin experiments with a simple circular model, whereby the circle represents an underlying submanifold to be respected, we find that the use of a VAE ensures that a posteriori ensemble members lie close to the manifold containing the truth. Furthermore, online updating of the VAE is necessary and achievable when this manifold varies in time, that is, when it is non‐stationary. We demonstrate that introducing an additional second latent space for the observational innovations improves robustness against detrimental effects of non‐Gaussianity and bias in the observational errors but lessens the performance slightly if observational errors are strictly Gaussian.

Developing operational data assimilation systems for sea-ice models is challenging, especially using a variational approach due to the absence of adjoint models. NeXtSIM, a sea-ice model based on a brittle rheology paradigm, enables high-fidelity simulations of sea-ice dynamics at mesoscale resolution (∼10 km) but lacks an adjoint. By training a neural network as an Arctic-wide emulator for sea-ice thickness based on mesoscale simulations with neXtSIM, we gain access to an adjoint. Building on this emulator and its adjoint, we introduce a four-dimensional variational (4D–Var) data assimilation system to correct the emulator's bias and to better position the marginal ice zone (MIZ). Firstly, we perform twin experiments to demonstrate the capabilities of this 4D–Var system and to evaluate two approximations of the background covariance matrix. These twin experiments demonstrate that the assimilation improves the positioning of the MIZ and enhances the forecast quality, achieving an average reduction in sea-ice thickness root-mean-squared error of 0.8 m compared to the free run. Secondly, we assimilate real CS2SMOS satellite retrievals with this system. While the assimilation of these rather smooth retrievals amplifies the loss of small-scale information in our system, it effectively corrects the forecast bias. The forecasts of our 4D–Var system achieve a similar performance as the operational sea-ice forecasting system neXtSIM-F. These results pave the way to the use of deep learning-based emulators for 4D–Var systems to improve sea-ice modeling.