Yumeng Chen

The Parallel Data Assimilation Framework (PDAF) is a widely used open-source software for data assimilation (DA) with complex, high-dimensional Earth system models and other applications for research and operational use. PDAF is structured to provide a framework for ensemble integrations and to assimilate observations. For the DA, it provides ensemble Kalman filters and smoothers, nonlinear ensemble DA methods, and 3D variational methods. PDAF's observation module interface (PDAF-OMI) further provides a system for structured observation handling, enabling the management of large numbers of different observation types. With the recent upgrade to version 3, PDAF underwent significant code modernization and functionality enhancements to unify more than 20 years of developments since its first release. This study provides a comprehensive overview of PDAF's functionality and concepts, including the new features introduced with the major revision. These are, in particular, a new universal interface that allows for the application of any ensemble filter and smoother method, or any 3D variational method, without changes to the source code. Further, a class of ensemble Kalman filters that processes observations serially, new diagnostics for the ensemble and observations, model-independent support for incremental analysis updating (IAU), and an explicit mode for file-based offline coupling between the model and the DA were added in the major revision. PDAF can apply the same DA methods to idealized toy models as well as realistic high-dimensional models without recoding. This speeds up the development and testing of new DA methods. The implementation of a DA system with PDAF is performed utilizing a set of template files. These allow for the implementation in a very limited time, supported by extensive documentation, as is demonstrated by an example implementation with a toy model. For high-dimensional applications, PDAF allows developers to start the implementation of a new DA system with low complexity and to extend its functionality stepwise, enabling an easy start with DA. Couplings to more than 30 models have been implemented by the PDAF developers and the user community and many of them are publicly available. Further, PDAF's internal interface to DA methods provides a clear approach to add new algorithms that can leverage PDAF's framework functionality for observations, localization, and state vector handling. This allows developers of DA methods to make their methods available to the PDAF community.

The state of the marine ecosystem can be estimated by a combination of numerical models and satellite observations through data assimilation (DA) methods. Satellite data representing phytoplankton chlorophyll are typically used in operational marine ecosystem prediction. These data are derived from ocean colour measured by optical satellite observations. Recently a new phytoplankton carbon product, from the ESA funded BICEP project made available from the UK CEDA Archive, has been derived through a novel processing of ocean colour. This novel carbon product captures aspects of phytoplankton biomass composition that chlorophyll alone does not represent. Here, we investigate the effects of assimilating the new carbon product on the modelling of the marine ecosystem. The investigation is carried out in a newly developed global ensemble DA system for the marine ecosystem using a coupled ocean-biogeochemistry model, NEMO-MEDUSA, and the Parallel Data Assimilation Framework. With the ensemble DA system, the evaluation can take the time-dependent uncertainty of the marine ecosystem and the reliability of the ensemble into account. We demonstrate that, compared with assimilating only the chlorophyll product, which may degrade the seasonality of phytoplankton carbon, assimilating the new carbon product can provide different patterns of adjustment and seasonal anomalies in phytoplankton concentrations, surface pCO ₂ , and oxygen. Our findings reveal that simultaneously assimilating both phytoplankton chlorophyll and carbon products in a complex marine ecosystem yields more balanced estimates of phytoplankton biomass than assimilating a single phytoplankton product.

The Parallel Data Assimilation Framework (PDAF) is a widely used open-source software for data assimilation (DA) with complex, high-dimensional Earth system models and other applications for research and operational use. PDAF is structured to provide a framework for ensemble integrations and to assimilate observations. For the DA, it provides ensemble Kalman filters and smoothers, nonlinear ensemble DA methods, and 3D variational methods. PDAF's observation module interface (PDAF-OMI) further provides a system for structured observation handling, enabling the management of large numbers of different observation types. With the recent upgrade to version 3, PDAF underwent significant code modernization and functionality enhancements to unify more than 20 years of developments since its first release. This study provides a comprehensive overview of PDAF's functionality and concepts, including the new features introduced with the major revision. These are, in particular, a new universal interface that allows for the application of any ensemble filter and smoother method, or any 3D variational method, without changes to the source code. Further, a class of ensemble Kalman filters that processes observations serially, new diagnostics for the ensemble and observations, model-independent support for incremental analysis updating (IAU), and an explicit mode for file-based offline coupling between the model and the DA were added in the major revision. PDAF can apply the same DA methods to idealized toy models as well as realistic high-dimensional models without recoding. This speeds up the development and testing of new DA methods. The implementation of a DA system with PDAF is performed utilizing a set of template files. These allow for the implementation in a very limited time, supported by extensive documentation, as is demonstrated by an example implementation with a toy model. For high-dimensional applications, PDAF allows developers to start the implementation of a new DA system with low complexity and to extend its functionality stepwise, enabling an easy start with DA. Couplings to more than 30 models have been implemented by the PDAF developers and the user community and many of them are publicly available. Further, PDAF's internal interface to DA methods provides a clear approach to add new algorithms that can leverage PDAF's framework functionality for observations, localization, and state vector handling. This allows developers of DA methods to make their methods available to the PDAF community.

The state of the marine ecosystem can be estimated by a combination of numerical models and satellite observations through data assimilation (DA) methods. Satellite data representing phytoplankton chlorophyll are typically used in operational marine ecosystem prediction. These data are derived from ocean colour from optical satellite observations. Recently a novel phytoplankton carbon product, from the ESA funded BICEP project available from the UK CEDA Archive, has been derived through an alternate processing of ocean colour. With the novel carbon product, the phytoplankton biomass is represented more directly than relying on the chlorophyll. Here, we investigate the effects of assimilating the new carbon product on the modelling of the marine ecosystem. The investigation is carried out in a newly developed global ensemble DA system for the marine ecosystem using a coupled ocean-biogeochemistry model, NEMO-MEDUSA, and the Parallel Data Assimilation Framework. With the ensemble DA system, the evaluation can take the time-dependent uncertainty of the marine ecosystem and the reliability of the ensemble into account. We demonstrate that, compared with solely assimilating chlorophyll product, with the new carbon product the DA can provide different patterns of adjustments in the phytoplankton concentration and seasonal anomalies. Our findings reveal that simultaneously assimilating both phytoplankton chlorophyll and carbon products in a complex marine ecosystem yields more accurate and balanced estimates of phytoplankton biomass than assimilating a single phytoplankton product.

The state of the marine ecosystem can be estimated by a combination of numerical models and satellite observations through data assimilation (DA) methods. Satellite data representing phytoplankton chlorophyll are typically used in operational marine ecosystem prediction. These data are derived from ocean colour from optical satellite observations. Recently a novel phytoplankton carbon product, from the ESA funded BICEP project available from the UK CEDA Archive, has been derived through an alternate processing of ocean colour. With the novel carbon product, the phytoplankton biomass is represented more directly than relying on the chlorophyll. Here, we investigate the effects of assimilating the new carbon product on the modelling of the marine ecosystem. The investigation is carried out in a newly developed global ensemble DA system for the marine ecosystem using a coupled ocean-biogeochemistry model, NEMO-MEDUSA, and the Parallel Data Assimilation Framework. With the ensemble DA system, the evaluation can take the time-dependent uncertainty of the marine ecosystem and the reliability of the ensemble into account. We demonstrate that, compared with solely assimilating chlorophyll product, with the new carbon product the DA can provide different patterns of adjustments in the phytoplankton concentration and seasonal anomalies. Our findings reveal that simultaneously assimilating both phytoplankton chlorophyll and carbon products in a complex marine ecosystem yields more accurate and balanced estimates of phytoplankton biomass than assimilating a single phytoplankton product.

PDAF is open-source software (https://pdaf.awi.de) providing a unified data assimilation framework for all data assimilation applications throughout the Earth system and beyond. PDAF is already coupled to a wide range of models, including all Earth system components, and is widely used for research and operational applications. With well-defined interfaces and modularization motivated by object-oriented programming, PDAF separates the forecast model, the observation handling, and the data assimilation algorithms. This structure ensures separation of concerns and allows domain experts to perform further developments of each component independently without interfering with each other. PDAF is further designed to make the coupling to models, online in memory or offline using disk files, particularly easy so that a new assimilation system can be built quickly. PDAF was recently upgraded to the new major revision 3.0. In PDAF V3, the code was modernized and restructured simplifying the procedure to add further data assimilation algorithms. New features are supported including model-agnostic incremental analysis updates, new diagnostics for observations and ensembles, and the ensemble square root filter (EnsRF) and ensemble adjustment Kalman filter (EAKF). With this, PDAF now provides the full range of algorithms from domain-localized ensemble filters and smoothers to Kalman filters with serial observation processing, particle and hybrid Kalman-nonlinear filters, and 3-dimensional variational data assimilation methods. Existing users can switch to PDAF V3 with minimal effort, while a new universal interface supporting all filters is recommended for new users. The Python-interface, pyPDAF, further allows the full implementation of an assimilation program in Python, leveraging the functionality and performance provided by PDAF. We will provide an overview of PDAF and the novelties of version 3.0.

The state of the marine ecosystem can be estimated by a combination of numerical models and satellite observations through data assimilation (DA) methods. Satellite data representing phytoplankton chlorophyll are typically used in operational marine ecosystem prediction. These data are derived from ocean colour from optical satellite observations. Recently a novel phytoplankton carbon product, from the ESA funded BICEP project available from the UK CEDA Archive, has been derived through an alternate processing of ocean colour. With the novel carbon product, the phytoplankton biomass is represented more directly than relying on the chlorophyll. Here, we investigate the effects of assimilating the new carbon product on the modelling of the marine ecosystem. The investigation is carried out in a newly developed global ensemble DA system for the marine ecosystem using a coupled ocean-biogeochemistry model, NEMO-MEDUSA, and the Parallel Data Assimilation Framework. With the ensemble DA system, the evaluation can take the time-dependent uncertainty of the marine ecosystem and the reliability of the ensemble into account. We demonstrate that, compared with solely assimilating chlorophyll product, with the new carbon product the DA can provide different patterns of adjustments in the phytoplankton concentration and seasonal anomalies. Our findings reveal that simultaneously assimilating both phytoplankton chlorophyll and carbon products in a complex marine ecosystem yields more accurate and balanced estimates of phytoplankton biomass than assimilating a single phytoplankton product.

Popular (ensemble) Kalman filter data assimilation (DA) approaches assume that the errors in both the a priori estimate of the state and the observations are Gaussian. For constrained variables, for example, sea‐ice concentration or stress, such an assumption does not hold. The variational autoencoder (VAE) is a machine‐learning (ML) technique that allows us to map an arbitrary distribution to/from a latent space in which the distribution is supposedly closer to a Gaussian. We propose a novel hybrid DA–ML approach in which VAEs are incorporated in the DA procedure. Specifically, we introduce a variant of the popular ensemble transform Kalman filter (ETKF) in which the analysis is applied in the latent space of a single VAE or a pair of VAEs. In twin experiments with a simple circular model, whereby the circle represents an underlying submanifold to be respected, we find that the use of a VAE ensures that a posteriori ensemble members lie close to the manifold containing the truth. Furthermore, online updating of the VAE is necessary and achievable when this manifold varies in time, that is, when it is non‐stationary. We demonstrate that introducing an additional second latent space for the observational innovations improves robustness against detrimental effects of non‐Gaussianity and bias in the observational errors but lessens the performance slightly if observational errors are strictly Gaussian.