Martina Stockhause

As Earth system models (ESMs) grow in complexity and in volume of output data, there is an increasing need for rapid, comprehensive evaluation of their scientific performance. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require expeditious response for model analyses designed to inform and drive integrated Earth system assessments. To meet this challenge, the Rapid Evaluation Framework (REF), a community-driven platform for benchmarking and performance assessment of ESMs, was designed and developed. The initial implementation of the REF, constructed to meet the near-term needs of the CMIP7 Assessment Fast Track, builds upon four disparate community evaluation and benchmarking tools that are coupled together using the Coordinated Model Evaluation Capabilities (CMEC) framework. The REF runs within a containerized workflow for portability and reproducibility and is aimed at generating and organizing diagnostics covering a variety of model variables. The REF leverages well documented observational datasets to provide assessments of model fidelity across a collection of diagnostics. All diagnostics were identified and selected with community involvement and consultation. Operational integration with the Earth System Grid Federation (ESGF) will permit automated execution of the REF for selected diagnostics as soon as model output data are published on ESGF by the originating modeling centers. The REF is designed to be portable across a range of current computational platforms to facilitate use by modeling centers for assessing the evolution of model versions or gauging the relative performance of CMIP simulations before being published on ESGF. When integrated into production simulation workflows, results from the REF provide immediate quantitative feedback that allows model developers and scientists to quickly identify model biases and performance issues. After the REF is released to the community, its subsequent development and support will be prioritized by an international consortium of scientists and engineers, enabling a broader impact across Earth science disciplines. For instance, the REF will facilitate improvements to models and will enhance confidence in model projections through process-based selection of models based on their performance with respect to observations. Production of reproducible diagnostics and community-based assessments are key features of the REF. Furthermore, providing interoperability with existing evaluation packages assures that contributions from previous community efforts will be available for use in future model intercomparison projects.

The FAIR principles — Findable, Accessible, Interoperable, and Reusable — underpin Open Science but does not fully ensure the long-term usability of interactive data services like the IPCC WGI Interactive Atlas. Drawing on lessons learned from developing and operating the Interactive Atlas, this presentation explores the challenges of sustaining such services, which rely not only on FAIR-compliant data and software but also on continuous stewardship, infrastructure maintenance, and institutional commitment.Scientific quality and transparency of the Interactive Atlas are supported through expert assessment by the IPCC authors, provenance documentation, and Complex Citation, which combine the attribution of credit for assessed digital objects with the traceability of digital IPCC results. Yet, sustaining reliability requires ongoing stewardship of both data and software to prevent degradation and preserve reproducibility. Addressing these needs demands joint efforts of the IPCC Data Distribution Centre (DDC) Partners to maintain data, documentation, and interactive components for a diverse user community. FAIR alone is not enough — long-term data preservation and infrastructure maintenance are essential to ensure the sustainability and trustworthiness of interactive data services in Earth system science.By reflecting on both the successes and limitations of the Interactive Atlas, this contribution offers insights relevant to other Earth system science communities developing interactive or service-oriented data products. These approaches are also applicable to fields beyond Earth system science.

As Earth system models (ESMs) grow in complexity and in volumes of output data, there is an increasing need for rapid, comprehensive evaluation of their scientific performance. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require expeditious response for model analyses designed to inform and drive integrated Earth system assessments. To meet this challenge, the Rapid Evaluation Framework (REF), a community-driven platform for benchmarking and performance assessment of ESMs, was designed and developed. The initial implementation of the REF, constructed to meet the near-term needs of the CMIP7 Assessment Fast Track, builds upon community evaluation and benchmarking tools. The REF runs within a containerized workflow for portability and reproducibility and is aimed at generating and organizing diagnostics covering a variety of model variables. The REF leverages best-available observational datasets to provide assessments of model fidelity across a collection of diagnostics. All diagnostics were identified and finally selected with community involvement and consultation. Operational integration with the Earth System Grid Federation (ESGF) will permit automated execution of the REF for specific diagnostics as soon as model data are published on ESGF by the originating modelling centres. The REF is designed to be portable across a range of current computational platforms to facilitate use by modelling centres for assessing the evolution of model versions or gauging the relative performance of CMIP simulations before being published on ESGF. When integrated into production simulation workflows, results from the REF provide immediate quantitative feedback that allows model developers and scientists to quickly identify model biases and performance issues. After the REF is released to the community, its subsequent development and support will be prioritized by an international consortium of scientists and engineers, enabling a broader impact across Earth science disciplines. For instance, the REF will facilitate improvements to models and reductions in uncertainties for projections since ESMs are the main tool for studying the global Earth system. Production of reproducible diagnostics and community-based assessments are the key features of the REF that help to inform mitigation and adaptation policies.

As Earth system models (ESMs) grow in complexity and in volumes of output data, there is an increasing need for rapid, comprehensive evaluation of their scientific performance. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require expeditious response for model analyses designed to inform and drive integrated Earth system assessments. To meet this challenge, the Rapid Evaluation Framework (REF), a community-driven platform for benchmarking and performance assessment of ESMs, was designed and developed. The initial implementation of the REF, constructed to meet the near-term needs of the CMIP7 Assessment Fast Track, builds upon community evaluation and benchmarking tools. The REF runs within a containerized workflow for portability and reproducibility and is aimed at generating and organizing diagnostics covering a variety of model variables. The REF leverages best-available observational datasets to provide assessments of model fidelity across a collection of diagnostics. All diagnostics were identified and finally selected with community involvement and consultation. Operational integration with the Earth System Grid Federation (ESGF) will permit automated execution of the REF for specific diagnostics as soon as model data are published on ESGF by the originating modelling centres. The REF is designed to be portable across a range of current computational platforms to facilitate use by modelling centres for assessing the evolution of model versions or gauging the relative performance of CMIP simulations before being published on ESGF. When integrated into production simulation workflows, results from the REF provide immediate quantitative feedback that allows model developers and scientists to quickly identify model biases and performance issues. After the REF is released to the community, its subsequent development and support will be prioritized by an international consortium of scientists and engineers, enabling a broader impact across Earth science disciplines. For instance, the REF will facilitate improvements to models and reductions in uncertainties for projections since ESMs are the main tool for studying the global Earth system. Production of reproducible diagnostics and community-based assessments are the key features of the REF that help to inform mitigation and adaptation policies.

As Earth system models (ESMs) grow in complexity and in volumes of output data, there is an increasing need for rapid, comprehensive evaluation of their scientific performance. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require expeditious response for model analyses designed to inform and drive integrated Earth system assessments. To meet this challenge, the Rapid Evaluation Framework (REF), a community-driven platform for benchmarking and performance assessment of ESMs, was designed and developed. The initial implementation of the REF, constructed to meet the near-term needs of the CMIP7 Assessment Fast Track, builds upon community evaluation and benchmarking tools. The REF runs within a containerized workflow for portability and reproducibility and is aimed at generating and organizing diagnostics covering a variety of model variables. The REF leverages best-available observational datasets to provide assessments of model fidelity across a collection of diagnostics. All diagnostics were identified and finally selected with community involvement and consultation. Operational integration with the Earth System Grid Federation (ESGF) will permit automated execution of the REF for specific diagnostics as soon as model data are published on ESGF by the originating modelling centres. The REF is designed to be portable across a range of current computational platforms to facilitate use by modelling centres for assessing the evolution of model versions or gauging the relative performance of CMIP simulations before being published on ESGF. When integrated into production simulation workflows, results from the REF provide immediate quantitative feedback that allows model developers and scientists to quickly identify model biases and performance issues. After the REF is released to the community, its subsequent development and support will be prioritized by an international consortium of scientists and engineers, enabling a broader impact across Earth science disciplines. For instance, the REF will facilitate improvements to models and reductions in uncertainties for projections since ESMs are the main tool for studying the global Earth system. Production of reproducible diagnostics and community-based assessments are the key features of the REF that help to inform mitigation and adaptation policies.

The Baseline Climate Variables for Earth System Modelling (ESM-BCVs) are defined as a list of 135 variables which have high utility for the evaluation and exploitation of climate simulations. The list reflects the most frequently used elements of the Coupled Model Intercomparison Project Phase 6 (CMIP6) archive. Successive phases of CMIP have supported strong results in science and substantially influence international climate policy formulation. This paper responds to both interest in exploiting CMIP data standards in a broader range of climate modelling activities and a need to achieve greater clarity about the significance and intention of variables in the CMIP Data Request. As Earth system modelling archives grow in scale and complexity, there are emerging problems associated with weak standardisation at the variable collection level. That is, there are good standards covering how specific variables should be archived, but this paper fills a gap in the standardisation of which variables should be archived. The ESM-BCV list is intended as a resource for ESM intercomparison projects (MIPs) developing requests to enable greater consistency among MIPs and as a reference for modelling centres to enhance consistency within MIPs. Provisional planning for the CMIP7 Data Request exploits the ESM-BCVs as a core element. The baseline variable list includes 98 variables which have modest or minor data volume footprints and could be generated systematically when simulations are produced and archived for exploitation by the World Climate Research Programme (WCRP) community. A further 35 variables are classed as “high volume” and are only suitable for production when the resource implications are justified.

The introduction of FAIR data practices into the assessment process of the Intergovernmental Panel on Climate Change (IPCC) enhances the transparency of the Assessment Reports (ARs). The approach focuses on the figure creation process and the reproducibility of figures within the reports. The process of creating figures for IPCC reports makes it particularly challenging to credit all contributors. A single figure can incorporate data from observations, models, and publications, originating from numerous sources and generated by multiple authors, each contributing to different parts of the figure. Additionally, it is not always straightforward to include all those involved in the creation process, as the final data (used to produce the figure) often result from assessments and post-processing of other data (input or intermediate data). Among the plans and recommendations for the current Seventh Assessment cycle is the use of the Complex Citation approach (Stockhause et al., 2024), which has been developed in the RDA Complex Citation Working Group (Agarwal et al., 2024).Some chapters of the ARs use established community frameworks for the generation of their figures. These are currently harmonized in the CMIP Rapid Evaluation Framework (REF) project for the AR7 Fast Track simulations. Others use interactive and shareable Jupyter notebooks. The IPCC needs to support and harmonize both ways of working. Further, the authors’ additional effort to meet the metadata and data requirements needs to be minimized. Therefore, the collaboration with both projects, REF and Jupyter, is planned to offer broad support for the ways of working of a large number of authors. This contribution analyzes how the Complex Citation approach, with the aims of giving credit to input providers and enabling the traceability of all digital objects used to create figures, can be implemented in REF and Jupyter and integrated in the AR7 process to support the IPCC’s FAIR and open data approach.References:Agarwal, D., Ayliffe, J., J. H. Buck, J., Damerow, J., Parton, G., Stall, S., Stockhause, M., & Wyborn, L. (2024). Complex Citation Working Group Recommendation. Zenodo. https://doi.org/10.5281/zenodo.14106602 Stockhause M, Huard D, Al Khourdajie A, Gutiérrez JM, Kawamiya M, Klutse NAB, et al. (2024) Implementing FAIR data principles in the IPCC seventh assessment cycle: Lessons learned and future prospects. PLOS Clim 3(12): e0000533. https://doi.org/10.1371/journal.pclm.0000533 Links:REF project: https://wcrp-cmip.org/cmip7/rapid-evaluation-framework/ Jupyter project: https://jupyter.org/

An ongoing challenge relevant to most research disciplines is the difficulty in citing 100+ digital objects such as datasets, software, samples, and images. Journals require authors to place citations over some set limit into supplemental information, where individual citations are not properly indexed, not linked to the manuscript, nor tracked accurately. Citing these research products is critical to enable transparent and reproducible research and for researchers, institutions, and project managers to trace citation, get appropriate credit, and report impact to funders. Open Science practices encourage providing proper attribution for the digital objects that support research findings and outcomes. Journals commonly redirect authors with many digital object citations to move those to the supplemental information where they are not indexed. This means: Creators of these digital objects do not get attribution and credit for their contribution to the scholarly literature Funders cannot measure use, impact and derived value from these digital objects Machine-actionable transparency is not possible. And over time, the supplement has a high probability of not being maintained by the publisher. We need to develop a scalable citation implementation strategy to enable open transparent and traceable research, which allows integration into common citation/impact metrics The findings of the Research Data Alliance (RDA) Complex Citations Working Group have produced key requirements (R1 - R10) for Complex Citation Objects (CCOs) to achieve our goals. In summary: CCOs capture enough detail to ensure proper credit, traceability, and transparency of cited materials (R1), supporting machine-actionable attribution for each referenced object (R2). CCOs do not accrue credit themselves but simply list data and digital identifiers that require citation tracking (R3). CCOs are stable, identifiable, versioned, resolvable, and persistent (R4, R5). CCOs use standardized structures, limited to two PID graph levels, with a strong preference to utilize persistent identifiers (R6, R6.1, R7). CCOs remain open, accessible, and flexible for various use cases, with an open license, and sufficient metadata (R8-R10). The full recommendations were published ahead of a presentation at the last RDA plenary session (Agarwal et al. 2024). The recommendations were based on use cases that identified the roles and responsibilities of the Complex Citation Workflow Actors necessary for the Complex Citation Objects (CCOs) to be used in practice. The Complex Citations Working Group is moving to a new phase where the recommendations need to be tested, evaluated and proven. To this end we are keen to inspire collaboration through new use cases, pilot implementations, to include repositories, journals, indexers and researchers to develop a new project and entrain more communities to take this work forward. Reference: Agarwal, D., Ayliffe, J., J. H. Buck, J., Damerow, J., Parton, G., Stall, S., Stockhause, M., & Wyborn, L. (2024). Complex Citation Working Group Recommendation (Version 1). Zenodo. https://doi.org/10.5281/zenodo.14106603

Intergovernmental Panel on Climate Change (IPCC) authors of assessment reports (ARs) and special reports (SRs) use a huge volume of input data, generate a great deal of intermediate data in the process, and produce a large amount of final data for figures and annexes in the published reports. In previous assessment cycles before the Sixth Assessment Report (AR6), only a limited amount of IPCC data were archived and made publicly available. There was great progress in the AR6, but many critical data sets were still not properly curated. This resulted in a data rescue effort during the transition from AR6 to AR7, supported by the IPCC and government fundings. The challenges encountered during the data rescue effort included missing or lost data after the report publication, missing data licensing agreements, version control issues, and missing data quality assurance/quality control (QA/QC) so that some data did not match the published figures. Addressing these issues demanded significantly more resources than the regular process to track, retrieve, archive, and resolve the legal and technical issues.In the Seventh Assessment Report (AR7), IPCC progressively promotes the FAIR data principles (Findable, Accessible, Interoperable, and Reusable) through the IPCC Task Group on Data Support for Climate Change Assessments (TG-Data) and the Data Distribution Centre (DDC) (1, 2). The Working Group Technical Support Units (TSUs) have also designated data specialists in the TG-Data (3). This provides opportunities to support authors in implementing open and FAIR data in IPCC AR7. For example, in Chapter 2 of the Special Report on Climate Change and Cities (SRCities), there is an area of focus on “Data, information, tools accessibility/availability/usability/transparency" (4). By collaborating the TSUs and DDC can provide a coordinated approach that supports authors with training and tools on data workflow, metadata schema, data provenance, licensing and citation, persistent identifiers, etc., to improve the data curation process and to avoid the issues encountered in previous cycles.References:1. Intergovernmental Panel on Climate Change. (2023). TG-Data Recommendations for AR7 (1.0). Zenodo. https://doi.org/10.5281/zenodo.10059282 2. Stockhause M, Huard D, Al Khourdajie A, Gutiérrez JM, Kawamiya M, Klutse NAB, Krey V, Milward D, Okem AE, Pirani A, Sitz LA, Solman SA, Spinuso A, Xing X. (2024). Implementing FAIR data principles in the IPCC seventh assessment cycle: Lessons learned and future prospects. PLOS Climate 3(12): e0000533. https://doi.org/10.1371/journal.pclm.0000533 3. https://www.ipcc.ch/data/ (2025) 4. IPCC Special Report on Climate Change and Cities (SRCities) report outline. (2024). https://www.ipcc.ch/site/assets/uploads/2024/08/IPCC-61_decisions-adopted-by-the-Panel.pdf

The CMIP6 project was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a long history, extending back four decades. CMIP has captivated and engaged a broad, growing community focused on improving our climate understanding. It has anchored our ability to quantify and attribute the drivers and responses of the observed climate changes we are experiencing today. The project's profound impact has been achieved by combining the latest climate science and technology. This has enabled the production of latest-generation climate simulations and the dissemination of their output, which has seen increased community attention in every successive phase. The review emphasizes the pragmatics of progressively scaling up efforts, the evolution of how the MIPs were implemented, and the coordinated efforts to establish a minimal infrastructure to make that possible, most recently delivering CMIP6.