Alessandro Spinuso
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Alessandro Spinuso has a total of 64 co-authors. Here are the top co-authors by number of shared publications:
- Christian Pagé
CERFACS
6 shared publications - Daniele Bailo
Istituto Nazionale di Geofisica e Vulcanologia, Roma, Italy
4 shared publications Rossana Paciello
3 shared publications- Wim Som de Cerff
Royal Netherlands Meteorological Institute
3 shared publications Ian van der Neut
2 shared publications- Irakalis Angelos Klampanos
NCSRD
2 shared publications
Latest Publications
- Open Access
- 22 June 2026
- Published by Copernicus GmbH
Reproducible dual-path cloud-native architecture for working with crowdsourced weather observations
Integrating crowdsourced weather observations into national meteorological service (NMS) infrastructures remains a complex challenge, driven by heterogeneous data formats, inconsistent access methods, and the need for reproducible workflows across research and operational environments. These obstacles often lead to duplicated effort and hinder the transition from exploratory analysis to production‑grade applications, especially when working with large‑scale geospatial datasets that demand modern Python‑based, cloud‑native, and GIS‑enabled processing workflows. To address this, we developed a proof-of-concept (PoC) with three connected components: open-source virtual research environment (i.e. SWIRRL), a spatial processing service (i.e. GRID), and an internal data platform (i.e. Party Platform) enabling data analysis directly in the cloud infrastructure. SWIRRL is a framework for VREs that supports JupyterLab-based workspaces, in combination with other interactive visualization tools and analysis workflows. It supports the automated collection of data and the execution of workflows that populate and transform those workspaces. Provenance is captured using standard provenance technologies allowing users to trace changes in data and tools. To support collaboration and Open Science, SWIRRL offers snapshot capabilities that allow users to publish their notebooks as a reproducible Jupyter Binder repository on GitHub. GRID is an internal KNMI development intended to encapsulate well-consolidated gridding methods used for current products and services. GRID exposes interpolation methods to convert datapoints into a continuous field and create contours lines based on the continuous field. The usage of GRID implies that the identical processing logic is applied across environments, hence aligning research and operational outputs.Party Platform is a serverless data platform that uses standard SQL. This platform provides a uniform interface to heterogeneous datasets, simplifying integration and ensuring reproducible access across research and operational contexts. Cloud-native architecture and Python integration allow rapid scaling of queries and processing tasks without impacting operational stability. The novelty of our approach lies in combining serverless, SQL-accessible data with reusable spatial processing services. Together, these components form a dual-path architecture that supports both research and operations on shared infrastructure while remaining logically isolated. This design enables rapid experimentation on production-grade infrastructure without compromising operational stability, hence closing the gap between prototyping and deployment. Future work will add full PROV-O provenance for all GRID processing steps, delivering [or improving] end-to-end transparency. Taken as a whole, this approach shows how unifying serverless data access with reusable geospatial processing can transform the integration of crowdsourced observations and accelerate both research innovation and operational impact.
- 22 June 2026
- Published by Copernicus GmbH
Open AccessReproducible dual-path cloud-native architecture for working with crowdsourced weather observations
Integrating crowdsourced weather observations into national meteorological service (NMS) infrastructures remains a complex challenge, driven by heterogeneous data formats, inconsistent access methods, and the need for reproducible workflows across research and operational environments. These obstacles often lead to duplicated effort and hinder the transition from exploratory analysis to production‑grade applications, especially when working with large‑scale geospatial datasets that demand modern Python‑based, cloud‑native, and GIS‑enabled processing workflows. To address this, we developed a proof-of-concept (PoC) with three connected components: open-source virtual research environment (i.e. SWIRRL), a spatial processing service (i.e. GRID), and an internal data platform (i.e. Party Platform) enabling data analysis directly in the cloud infrastructure. SWIRRL is a framework for VREs that supports JupyterLab-based workspaces, in combination with other interactive visualization tools and analysis workflows. It supports the automated collection of data and the execution of workflows that populate and transform those workspaces. Provenance is captured using standard provenance technologies allowing users to trace changes in data and tools. To support collaboration and Open Science, SWIRRL offers snapshot capabilities that allow users to publish their notebooks as a reproducible Jupyter Binder repository on GitHub. GRID is an internal KNMI development intended to encapsulate well-consolidated gridding methods used for current products and services. GRID exposes interpolation methods to convert datapoints into a continuous field and create contours lines based on the continuous field. The usage of GRID implies that the identical processing logic is applied across environments, hence aligning research and operational outputs.Party Platform is a serverless data platform that uses standard SQL. This platform provides a uniform interface to heterogeneous datasets, simplifying integration and ensuring reproducible access across research and operational contexts. Cloud-native architecture and Python integration allow rapid scaling of queries and processing tasks without impacting operational stability. The novelty of our approach lies in combining serverless, SQL-accessible data with reusable spatial processing services. Together, these components form a dual-path architecture that supports both research and operations on shared infrastructure while remaining logically isolated. This design enables rapid experimentation on production-grade infrastructure without compromising operational stability, hence closing the gap between prototyping and deployment. Future work will add full PROV-O provenance for all GRID processing steps, delivering [or improving] end-to-end transparency. Taken as a whole, this approach shows how unifying serverless data access with reusable geospatial processing can transform the integration of crowdsourced observations and accelerate both research innovation and operational impact.
- Open Access
- 18 March 2025
- Published by Copernicus GmbH
Jupyter Notebooks in European Plate Observing System (EPOS)
The European Plate Observing System (EPOS) addresses the problem of homogeneous access to heterogeneous digital assets in geoscience within the European tectonic plate. EPOS is a European Research Infrastructure Consortium (ERIC) since 2018, with the goal of building long-term and sustainable infrastructure for solid Earth science. The EPOS Data Portal was launched into the operational phase in April 2023 and is introducing new ways for cross-disciplinary research, especially for data discovery. Currently the EPOS Data Portal, a metadata and semantic-driven system for integrating Data, Software and services, provides access to data and data products from ten different geoscientific areas: Seismology, Near Fault Observatories, GNSS Data and Products, Volcano Observations, Satellite Data, Geomagnetic Observations, Anthropogenic Hazards, Geological Information and Modelling, Multi-scale laboratories and Tsunami Research. The presentation shows the achievements of the EPOS community, demonstrates features of the Portal user interface and also the underlying architecture of the whole system and online processing environment. The IT system as such is shared as Open Source and is under continuous development and improvement, following the SDLC methodology: Shape Up.This presentation focuses on the integration of Jupyter Notebooks into EPOS through Virtual Research Environment (VRE) which allows advanced processing of datasets provided already through EPOS Data Portal. Examples of Jupyter Notebooks covering various scientific multidisciplinary use cases introduce typical data processing workflows and visualizations for efficient use of services collected through EPOS. Expansion of the EPOS-DCAT-AP metadata model by new entities for managing software components and its utilization opens new possibilities for data access and paves the road for integration into other e-infrastructures.
- 18 March 2025
- Published by Copernicus GmbH
Open AccessJupyter Notebooks in European Plate Observing System (EPOS)
The European Plate Observing System (EPOS) addresses the problem of homogeneous access to heterogeneous digital assets in geoscience within the European tectonic plate. EPOS is a European Research Infrastructure Consortium (ERIC) since 2018, with the goal of building long-term and sustainable infrastructure for solid Earth science. The EPOS Data Portal was launched into the operational phase in April 2023 and is introducing new ways for cross-disciplinary research, especially for data discovery. Currently the EPOS Data Portal, a metadata and semantic-driven system for integrating Data, Software and services, provides access to data and data products from ten different geoscientific areas: Seismology, Near Fault Observatories, GNSS Data and Products, Volcano Observations, Satellite Data, Geomagnetic Observations, Anthropogenic Hazards, Geological Information and Modelling, Multi-scale laboratories and Tsunami Research. The presentation shows the achievements of the EPOS community, demonstrates features of the Portal user interface and also the underlying architecture of the whole system and online processing environment. The IT system as such is shared as Open Source and is under continuous development and improvement, following the SDLC methodology: Shape Up.This presentation focuses on the integration of Jupyter Notebooks into EPOS through Virtual Research Environment (VRE) which allows advanced processing of datasets provided already through EPOS Data Portal. Examples of Jupyter Notebooks covering various scientific multidisciplinary use cases introduce typical data processing workflows and visualizations for efficient use of services collected through EPOS. Expansion of the EPOS-DCAT-AP metadata model by new entities for managing software components and its utilization opens new possibilities for data access and paves the road for integration into other e-infrastructures.
- Open Access
- 20 January 2025
- Published by Copernicus GmbH
How do the CMIP7 infrastructure plans support the implementation of the IPCC data guidelines?
The Intergovernmental Panel on Climate Change (IPCC) regularly assesses a wide range of research results related to climate change reaching from physical sciences to economic and social sciences to provide policy makers with options for combatting the challenges of climate change. The IPCC authors analyze data across multiple domains and from multiple sources.The IPCC data guidelines enhance the transparency of IPCC outcomes by ensuring that figure creation is traceable, citing input data and long-term preserving data and software. The related data and metadata requested from the authors includes detailed information on datasets used in every figure, for which standardized machine-accessible and -readable information needs to be supported by the input data providers.An example of an important input data provider is the Coupled Model Intercomparison Project (CMIP), which has continuously improved its standards and data infrastructure to keep track with the significant increase in the scale of the project over recent phases. CMIP defines a set of standards including vocabularies for controlled metadata fields, e.g. variable and experiment names, along with the data itself and its structure. A set of infrastructure services provide access to data, through the Earth System Grid Federation (ESGF), description of the climate models used and known errata, through ES-DOC services, and data citation information including data usage in derivative data sets and published papers where known. The contribution will discuss the diverse data-workflows of the IPCC authors and the ways that the CMIP infrastructure supports them. Authors access data from the primary data portals of the ESGF, but also from secondary data portals (Copernicus, Pangeo, Climate4Impact) or local data pools hosted by national institutions. The IPCC authors have faced a number of challenges, including accessing data citation and model description information together with the data, and in identifying new dataset versions with significant changes. With the IPCC’s plan to utilize provenance records in AR7 to gather all information requested by the IPCC data guidelines, machine-readable information accessible through the file headers becomes essential. The entire IPCC AR7 data workflow needs to be supported by tools: the figure creation by the IPCC authors, the report editing process by the TSU and the curation of the CMIP7 input data subset used and the intermediate and final datasets created by various DDC Partners, including bi-directional references between outputs. Virtual workspaces such as CEDA and DKRZ provided for the authors in AR6, which give access to their data pools and common software packages like the ESMValtool, can support the authors in the preparation of their figures and the provision of the requested documentation and provenance information. A dedicated Figure Manager will play a central role in managing the report figures and supporting the overall data workflow. Ultimately, lowering the burden for the authors, the TSU staff and the DDC Partners. This timely gathered information can then be analyzed and used for a harmonization of dataset version usage across the chapters and reports.
- 20 January 2025
- Published by Copernicus GmbH
Open AccessHow do the CMIP7 infrastructure plans support the implementation of the IPCC data guidelines?
The Intergovernmental Panel on Climate Change (IPCC) regularly assesses a wide range of research results related to climate change reaching from physical sciences to economic and social sciences to provide policy makers with options for combatting the challenges of climate change. The IPCC authors analyze data across multiple domains and from multiple sources.The IPCC data guidelines enhance the transparency of IPCC outcomes by ensuring that figure creation is traceable, citing input data and long-term preserving data and software. The related data and metadata requested from the authors includes detailed information on datasets used in every figure, for which standardized machine-accessible and -readable information needs to be supported by the input data providers.An example of an important input data provider is the Coupled Model Intercomparison Project (CMIP), which has continuously improved its standards and data infrastructure to keep track with the significant increase in the scale of the project over recent phases. CMIP defines a set of standards including vocabularies for controlled metadata fields, e.g. variable and experiment names, along with the data itself and its structure. A set of infrastructure services provide access to data, through the Earth System Grid Federation (ESGF), description of the climate models used and known errata, through ES-DOC services, and data citation information including data usage in derivative data sets and published papers where known. The contribution will discuss the diverse data-workflows of the IPCC authors and the ways that the CMIP infrastructure supports them. Authors access data from the primary data portals of the ESGF, but also from secondary data portals (Copernicus, Pangeo, Climate4Impact) or local data pools hosted by national institutions. The IPCC authors have faced a number of challenges, including accessing data citation and model description information together with the data, and in identifying new dataset versions with significant changes. With the IPCC’s plan to utilize provenance records in AR7 to gather all information requested by the IPCC data guidelines, machine-readable information accessible through the file headers becomes essential. The entire IPCC AR7 data workflow needs to be supported by tools: the figure creation by the IPCC authors, the report editing process by the TSU and the curation of the CMIP7 input data subset used and the intermediate and final datasets created by various DDC Partners, including bi-directional references between outputs. Virtual workspaces such as CEDA and DKRZ provided for the authors in AR6, which give access to their data pools and common software packages like the ESMValtool, can support the authors in the preparation of their figures and the provision of the requested documentation and provenance information. A dedicated Figure Manager will play a central role in managing the report figures and supporting the overall data workflow. Ultimately, lowering the burden for the authors, the TSU staff and the DDC Partners. This timely gathered information can then be analyzed and used for a harmonization of dataset version usage across the chapters and reports.
- Open Access
- 20 January 2025
- Published by Copernicus GmbH
The EPOS open source platform for multidisciplinary data integration and data analysis in solid Earth science
Established as a European Research Infrastructure Consortium (ERIC) in 2018, the European Plate Observing System (EPOS) Research Infrastructure represents a significant advancement in solid Earth sciences. Its aim is to harmonize and integrate data, services, and computational resources across diverse solid Earth science domains. These include Seismology, Near-Fault Observatories, GNSS Data and Products, Volcano Observations, Satellite Data, Geomagnetic Observations, Anthropogenic Hazards, Geological Information and Modeling, Multi-Scale Laboratories, Tsunami Research, each leveraging EPOS for the integration of domain specific data and services into a wider European multi-disciplinary context.The EPOS platform ( https://www.epos-eu.org/dataportal) provides access to harmonized and quality-controlled data from thematic solid Earth science services through over 250 interoperable multidisciplinary services. The platform adopts a microservice-based architecture serving RESTful APIs, ensuring seamless interoperability between thematic core services (TCS) and the integrated core services central hub (ICS-C). The ICS-C, as the central system underpinning the EPOS platform, enables interoperability by adopting a multidimensional approach using metadata, semantics, and web services. Released under a GPL license as open-source software (https://epos-eu.github.io/epos-open-source/), EPOS adheres to the FAIR Principles, fostering interdisciplinary collaboration and technological advancement in Earth sciences and beyond.In addition to data access, the EPOS platform also integrates complementary visualization tools and computational services. These Integrated Core Services - Distributed (ICS-D) enhance the user experience by simplifying complex interactions, offering functionalities like visualization, coding, and processing for data analysis, including machine learning applications.This presentation will explore how the EPOS platform facilitates the entire research data lifecycle, connecting integrated multidisciplinary data provision to remote data analysis environments. By leveraging third-party cloud and supercomputing facilities equipped with specialized APIs (eg. SWIRRL https://gitlab.com/KNMI-OSS/swirrl/swirrl-api), we will demonstrate how EPOS seamlessly integrates with external services for reproducible data analysis and visualization, relying on common workflows to gather and pre-preprocess the data. External service examples include Jupyter Notebooks developed by domain-specific communities, using which the users can immediately analyze and process the data online. This adaptability streamlines scientific research and also promotes data reusability and collaboration within the portal, showcasing the EPOS platform's role in advancing Earth sciences research.
- 20 January 2025
- Published by Copernicus GmbH
Open AccessThe EPOS open source platform for multidisciplinary data integration and data analysis in solid Earth science
Established as a European Research Infrastructure Consortium (ERIC) in 2018, the European Plate Observing System (EPOS) Research Infrastructure represents a significant advancement in solid Earth sciences. Its aim is to harmonize and integrate data, services, and computational resources across diverse solid Earth science domains. These include Seismology, Near-Fault Observatories, GNSS Data and Products, Volcano Observations, Satellite Data, Geomagnetic Observations, Anthropogenic Hazards, Geological Information and Modeling, Multi-Scale Laboratories, Tsunami Research, each leveraging EPOS for the integration of domain specific data and services into a wider European multi-disciplinary context.The EPOS platform ( https://www.epos-eu.org/dataportal) provides access to harmonized and quality-controlled data from thematic solid Earth science services through over 250 interoperable multidisciplinary services. The platform adopts a microservice-based architecture serving RESTful APIs, ensuring seamless interoperability between thematic core services (TCS) and the integrated core services central hub (ICS-C). The ICS-C, as the central system underpinning the EPOS platform, enables interoperability by adopting a multidimensional approach using metadata, semantics, and web services. Released under a GPL license as open-source software (https://epos-eu.github.io/epos-open-source/), EPOS adheres to the FAIR Principles, fostering interdisciplinary collaboration and technological advancement in Earth sciences and beyond.In addition to data access, the EPOS platform also integrates complementary visualization tools and computational services. These Integrated Core Services - Distributed (ICS-D) enhance the user experience by simplifying complex interactions, offering functionalities like visualization, coding, and processing for data analysis, including machine learning applications.This presentation will explore how the EPOS platform facilitates the entire research data lifecycle, connecting integrated multidisciplinary data provision to remote data analysis environments. By leveraging third-party cloud and supercomputing facilities equipped with specialized APIs (eg. SWIRRL https://gitlab.com/KNMI-OSS/swirrl/swirrl-api), we will demonstrate how EPOS seamlessly integrates with external services for reproducible data analysis and visualization, relying on common workflows to gather and pre-preprocess the data. External service examples include Jupyter Notebooks developed by domain-specific communities, using which the users can immediately analyze and process the data online. This adaptability streamlines scientific research and also promotes data reusability and collaboration within the portal, showcasing the EPOS platform's role in advancing Earth sciences research.
- Open Access
- 15 May 2023
- Published by Copernicus GmbH
Tools to support climate researchers for long-tail research data in a FAIR context
Doing high quality research involves complex workflows and intermediate datasets. An important part is also sharing of those datasets, software tools and workflows among researchers, and tracking provenance and lineage. It also needs to be stored in a citable permanent repository in order to be referenced in papers and reused subsequently by other researchers. Supporting this research data life cycle properly is a very challenging objective for research infrastructures. This is especially true with rapidly evolving technologies, sustainable funding problems and human expertise. In the climate research infrastructure, many efforts have been made to support end-users and long tail research. There is the basic data distribution, the ESGF data nodes, but this is to support mainly specialized researchers in climate science. This basic infrastructure implements quite strict standards to enable proper data sharing in the research community. This is far from FAIR compliance, but this has proven to be extremely beneficial for collaborative research. Of course, high level components and services can be built on top. This is not an easy task, and a layered approach is always better to hide the underlying complexity and also to prevent technology locking and too complex codes. One example is the IS-ENES C4I 2.0 platform (https://dev.climate4impact.eu/ ), a front-end that eases very much data access, and is acting like a bridge between the data nodes and computing services. The C4I platform provides a very much enhanced Jupyter-la like interface (SWIRRL), with many services to support sharing of data and common workflow for data staging and preprocessing, as well as the development of new analysis methods in a research context. Advanced tools that can calculate end-user products are also made available along with some example notebooks implementing popular workflows. One of these tools is icclim (https://github.com/cerfacs-globc/icclim), a python software package. C4I also includes high-level services such as on-the-fly inter-comparisons between climate simulations with ESMValTool (https://github.com/ESMValGroup/ESMValTool). All this work is also including large efforts to standardize and to become closer to FAIR for data, workflows and software.Another way of helping researchers is to pre-compute end-users products like climate indices. This is extremely useful for users because it can be really complex and time consuming to calculate those products. One example is to provide those users datasets of climate indices pre-computed on CMIP6 simulations would be very valuable for those users. Of course all specific needs cannot be taken into account but the most general ones can be fulfilled. The European Open Science Cloud (EOSC) is providing computing and storage resources through the EGI-ACE project, enabling the possibility to compute several climate indices. In this EGI-ACE Use Case, icclim will be used to compute 49 standard climate indices on a large number of CMIP6 simulations, starting with the most used ones. It could also be extended to ERA5 reanalysis, CORDEX and CMIP5 datasets.This project (IS-ENES3) has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N°824084.
- 15 May 2023
- Published by Copernicus GmbH
Open AccessTools to support climate researchers for long-tail research data in a FAIR context
Doing high quality research involves complex workflows and intermediate datasets. An important part is also sharing of those datasets, software tools and workflows among researchers, and tracking provenance and lineage. It also needs to be stored in a citable permanent repository in order to be referenced in papers and reused subsequently by other researchers. Supporting this research data life cycle properly is a very challenging objective for research infrastructures. This is especially true with rapidly evolving technologies, sustainable funding problems and human expertise. In the climate research infrastructure, many efforts have been made to support end-users and long tail research. There is the basic data distribution, the ESGF data nodes, but this is to support mainly specialized researchers in climate science. This basic infrastructure implements quite strict standards to enable proper data sharing in the research community. This is far from FAIR compliance, but this has proven to be extremely beneficial for collaborative research. Of course, high level components and services can be built on top. This is not an easy task, and a layered approach is always better to hide the underlying complexity and also to prevent technology locking and too complex codes. One example is the IS-ENES C4I 2.0 platform (https://dev.climate4impact.eu/ ), a front-end that eases very much data access, and is acting like a bridge between the data nodes and computing services. The C4I platform provides a very much enhanced Jupyter-la like interface (SWIRRL), with many services to support sharing of data and common workflow for data staging and preprocessing, as well as the development of new analysis methods in a research context. Advanced tools that can calculate end-user products are also made available along with some example notebooks implementing popular workflows. One of these tools is icclim (https://github.com/cerfacs-globc/icclim), a python software package. C4I also includes high-level services such as on-the-fly inter-comparisons between climate simulations with ESMValTool (https://github.com/ESMValGroup/ESMValTool). All this work is also including large efforts to standardize and to become closer to FAIR for data, workflows and software.Another way of helping researchers is to pre-compute end-users products like climate indices. This is extremely useful for users because it can be really complex and time consuming to calculate those products. One example is to provide those users datasets of climate indices pre-computed on CMIP6 simulations would be very valuable for those users. Of course all specific needs cannot be taken into account but the most general ones can be fulfilled. The European Open Science Cloud (EOSC) is providing computing and storage resources through the EGI-ACE project, enabling the possibility to compute several climate indices. In this EGI-ACE Use Case, icclim will be used to compute 49 standard climate indices on a large number of CMIP6 simulations, starting with the most used ones. It could also be extended to ERA5 reanalysis, CORDEX and CMIP5 datasets.This project (IS-ENES3) has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N°824084.
- Open Access
- 15 May 2023
- Published by Copernicus GmbH
Provenance powered microservices: a flexible and generic approach fostering reproducible research in Earth Science
Scientific progress requires research outputs to be reproducible, or at least persistently traceable and analysable for defects through time. This can be facilitated by coupling analysis tools that are already familiar to scientists, with reproducibility controls designed around common containerisation technologies and formats to represent metadata and provenance. Moreover, modern interactive tools for data analysis and visualisation, such as computational notebooks and visual analytics systems, are built to expose their functionalities through the Web. This facilitates the development of integrated solutions that are designed to support computational research with reproducibility in mind, and that, once deployed onto a Cloud infrastructure, benefit from operations that are securely managed and perform reliably. Such systems should be able to easily accommodate specific requirements concerning, for instance, the deployment of particular scientific software and the collection of tailored, yet comprehensive, provenance recordings about data and processes. By decoupling and generalising the description of the environment where a particular research took place from the underlying implementation, which may become obsolete through time, we obtain better chances to recollect relevant information for the retrospective analysis of a scientific product in the long term, enhancing preservation and reproducibility of results.In this contribution we illustrate how this is achievable via the adoption of microservice architectures combined with a provenance model that supports metadata standards and templating. We aim at empowering scientific data portals with Virtual Research Environments (VREs) and provenance services, that are programmatically controlled via high-level functions over the internet. Our system SWIRRL deals, on behalf of the clients, with the complexity of allocating the interactive services for the VREs on a Cloud platform. It runs staging and preprocessing workflows to gather and organise remote datasets, making them accessible collaboratively. We show how Provenance Services manage provenance records about the underlying environment, datasets and analysis workflows, and how these are exploited by researchers to control different reproducibility use cases. Our solutions are currently being implemented in more contexts in Earth Science. We will provide an overview on the progress of these efforts for the EPOS and IS-ENES research infrastructures, addressing solid earth and climate studies, respectively.Finally, although the reproducibility challenges can be tackled to a large extent by modern technology, this will be further consolidated and made interoperable via the implementation and uptake of the FDOs. To achieve this goal, it is fundamental to establish the conversation between engineers, data-stewards and researchers early in the process of delivering a scientific product. This fosters the definition and implementation of suitable best practices to be adopted by a particular research group. Scientific tools and repositories built around modern FAIR enabling resources can be incrementally refined thanks to this mediated exchange. We will briefly introduce success stories towards this goal in the context of the IPCC Assessment Reports.
- 15 May 2023
- Published by Copernicus GmbH
Open AccessProvenance powered microservices: a flexible and generic approach fostering reproducible research in Earth Science
Scientific progress requires research outputs to be reproducible, or at least persistently traceable and analysable for defects through time. This can be facilitated by coupling analysis tools that are already familiar to scientists, with reproducibility controls designed around common containerisation technologies and formats to represent metadata and provenance. Moreover, modern interactive tools for data analysis and visualisation, such as computational notebooks and visual analytics systems, are built to expose their functionalities through the Web. This facilitates the development of integrated solutions that are designed to support computational research with reproducibility in mind, and that, once deployed onto a Cloud infrastructure, benefit from operations that are securely managed and perform reliably. Such systems should be able to easily accommodate specific requirements concerning, for instance, the deployment of particular scientific software and the collection of tailored, yet comprehensive, provenance recordings about data and processes. By decoupling and generalising the description of the environment where a particular research took place from the underlying implementation, which may become obsolete through time, we obtain better chances to recollect relevant information for the retrospective analysis of a scientific product in the long term, enhancing preservation and reproducibility of results.In this contribution we illustrate how this is achievable via the adoption of microservice architectures combined with a provenance model that supports metadata standards and templating. We aim at empowering scientific data portals with Virtual Research Environments (VREs) and provenance services, that are programmatically controlled via high-level functions over the internet. Our system SWIRRL deals, on behalf of the clients, with the complexity of allocating the interactive services for the VREs on a Cloud platform. It runs staging and preprocessing workflows to gather and organise remote datasets, making them accessible collaboratively. We show how Provenance Services manage provenance records about the underlying environment, datasets and analysis workflows, and how these are exploited by researchers to control different reproducibility use cases. Our solutions are currently being implemented in more contexts in Earth Science. We will provide an overview on the progress of these efforts for the EPOS and IS-ENES research infrastructures, addressing solid earth and climate studies, respectively.Finally, although the reproducibility challenges can be tackled to a large extent by modern technology, this will be further consolidated and made interoperable via the implementation and uptake of the FDOs. To achieve this goal, it is fundamental to establish the conversation between engineers, data-stewards and researchers early in the process of delivering a scientific product. This fosters the definition and implementation of suitable best practices to be adopted by a particular research group. Scientific tools and repositories built around modern FAIR enabling resources can be incrementally refined thanks to this mediated exchange. We will briefly introduce success stories towards this goal in the context of the IPCC Assessment Reports.
- Open Access
- 29 April 2022
- Published by Instituto Nazionale di Geofisica e Vulcanologia, INGV in Annals of Geophysics
- Vol. 65 (2) , DM210
Data integration and FAIR data management in Solid Earth Science
Integrated use of multidisciplinary data is nowadays a recognized trend in scientific research, in particular in the domain of solid Earth science where the understanding of a physical process is improved and made complete by different types of measurements – for instance, ground acceleration, SAR imaging, crustal deformation – describing a physical phenomenon. FAIR principles are recognized as a means to foster data integration by providing a common set of criteria for building data stewardship systems for Open Science. However, the implementation of FAIR principles raises issues along dimensions like governance and legal beyond, of course, the technical one. In the latter, in particular, the development of FAIR data provision systems is often delegated to Research Infrastructures or data providers, with support in terms of metrics and best practices offered by cluster projects or dedicated initiatives. In the current work, we describe the approach to FAIR data management in the European Plate Observing System (EPOS), a distributed research infrastructure in the solid Earth science domain that includes more than 250 individual research infrastructures across 25 countries in Europe. We focus in particular on the technical aspects, but including also governance, policies and organizational elements, by describing the architecture of the EPOS delivery framework both from the organizational and technical point of view and by outlining the key principles used in the technical design. We describe how a combination of approaches, namely rich metadata and service-based systems design, are required to achieve data integration. We show the system architecture and the basic features of the EPOS data portal, that integrates data from more than 220 services in a FAIR way. The construction of such a portal was driven by the EPOS FAIR data management approach, that by defining a clear roadmap for compliance with the FAIR principles, produced a number of best practices and technical approaches for complying with the FAIR principles. Such a work, that spans over a decade but concentrates the key efforts in the last 5 years with the EPOS Implementation Phase project and the establishment of EPOS-ERIC, was carried out in synergy with other EU initiatives dealing with FAIR data. On the basis of the EPOS experience, future directions are outlined, emphasizing the need to provide i) FAIR reference architectures that can ease data practitioners and engineers from the domain communities to adopt FAIR principles and build FAIR data systems; ii) a FAIR data management framework addressing FAIR through the entire data lifecycle, including reproducibility and provenance; and iii) the extension of the FAIR principles to policies and governance dimensions. .
- 29 April 2022
- Published by Instituto Nazionale di Geofisica e Vulcanologia, INGV in Annals of Geophysics
- Vol. 65 (2) , DM210
Open AccessData integration and FAIR data management in Solid Earth Science
Integrated use of multidisciplinary data is nowadays a recognized trend in scientific research, in particular in the domain of solid Earth science where the understanding of a physical process is improved and made complete by different types of measurements – for instance, ground acceleration, SAR imaging, crustal deformation – describing a physical phenomenon. FAIR principles are recognized as a means to foster data integration by providing a common set of criteria for building data stewardship systems for Open Science. However, the implementation of FAIR principles raises issues along dimensions like governance and legal beyond, of course, the technical one. In the latter, in particular, the development of FAIR data provision systems is often delegated to Research Infrastructures or data providers, with support in terms of metrics and best practices offered by cluster projects or dedicated initiatives. In the current work, we describe the approach to FAIR data management in the European Plate Observing System (EPOS), a distributed research infrastructure in the solid Earth science domain that includes more than 250 individual research infrastructures across 25 countries in Europe. We focus in particular on the technical aspects, but including also governance, policies and organizational elements, by describing the architecture of the EPOS delivery framework both from the organizational and technical point of view and by outlining the key principles used in the technical design. We describe how a combination of approaches, namely rich metadata and service-based systems design, are required to achieve data integration. We show the system architecture and the basic features of the EPOS data portal, that integrates data from more than 220 services in a FAIR way. The construction of such a portal was driven by the EPOS FAIR data management approach, that by defining a clear roadmap for compliance with the FAIR principles, produced a number of best practices and technical approaches for complying with the FAIR principles. Such a work, that spans over a decade but concentrates the key efforts in the last 5 years with the EPOS Implementation Phase project and the establishment of EPOS-ERIC, was carried out in synergy with other EU initiatives dealing with FAIR data. On the basis of the EPOS experience, future directions are outlined, emphasizing the need to provide i) FAIR reference architectures that can ease data practitioners and engineers from the domain communities to adopt FAIR principles and build FAIR data systems; ii) a FAIR data management framework addressing FAIR through the entire data lifecycle, including reproducibility and provenance; and iii) the extension of the FAIR principles to policies and governance dimensions. .
- Open Access
- 1 April 2022
- Published by China Science Publishing & Media Ltd. in Data Intelligence
- Vol. 4 (2) , 243-258
SWIRRL. Managing Provenance-aware and Reproducible Workspaces
Modern interactive tools for data analysis and visualisation are designed to expose their functionalities as a service through the Web. We present in this paper a Web API (SWIRRL) that allows Virtual Research Environments (VREs) to easily integrate such tools in their websites and re-purpose them to their users. The API deals, on behalf of the clients, with the underlying complexity of allocating and managing resources within a target cloud platform. By combining storage and containerised services, offering analysis notebooks and other visualisation software, the API creates dedicated working sessions on-demand, which can be accessed collaboratively. Thanks to the API's support for workflow execution, SWIRRL workspaces can be automatically populated with data of interest collected from external data providers. The system keeps track of updates and changes affecting the data and the tools by adopting versioning and standard provenance technologies. Users are provided with interactive controls enabling traceability and recovery actions, including the possibility of creating executable snapshots of their environments. SWIRRL is built in cooperation with two research infrastructures in the field of solid earth science and climate data modeling. We report on the particular adoptions and use cases.
- 1 April 2022
- Published by China Science Publishing & Media Ltd. in Data Intelligence
- Vol. 4 (2) , 243-258
Open AccessSWIRRL. Managing Provenance-aware and Reproducible Workspaces
Modern interactive tools for data analysis and visualisation are designed to expose their functionalities as a service through the Web. We present in this paper a Web API (SWIRRL) that allows Virtual Research Environments (VREs) to easily integrate such tools in their websites and re-purpose them to their users. The API deals, on behalf of the clients, with the underlying complexity of allocating and managing resources within a target cloud platform. By combining storage and containerised services, offering analysis notebooks and other visualisation software, the API creates dedicated working sessions on-demand, which can be accessed collaboratively. Thanks to the API's support for workflow execution, SWIRRL workspaces can be automatically populated with data of interest collected from external data providers. The system keeps track of updates and changes affecting the data and the tools by adopting versioning and standard provenance technologies. Users are provided with interactive controls enabling traceability and recovery actions, including the possibility of creating executable snapshots of their environments. SWIRRL is built in cooperation with two research infrastructures in the field of solid earth science and climate data modeling. We report on the particular adoptions and use cases.
- Open Access
- 1 April 2022
- Published by China Science Publishing & Media Ltd. in Data Intelligence
- Vol. 4 (2) , 226-242
S-ProvFlow. Storing and Exploring Lineage Data as a Service
We present a set of configurable Web service and interactive tools, s-ProvFlow, for managing and exploiting records tracking data lineage during workflow runs. It facilitates detailed analysis of single executions. It helps users manage complex tasks by exposing the relationships between data, people, equipment and workflow runs intended to combine productively. Its logical model extends the PROV standard to precisely record parallel data-streaming applications. Its metadata handling encourages users to capture the application context by specifying how application attributes, often using standard vocabularies, should be added. These metadata records immediately help productivity as the interactive tools support their use in selection and bulk operations. Users rapidly appreciate the power of the encoded semantics as they reap the benefits. This improves the quality of provenance for users and management. Which in turn facilitates analysis of collections of runs, enabling users to manage results and validate procedures. It fosters reuse of data and methods and facilitates diagnostic investigations and optimisations. We present S-ProvFlow's use by scientists, research engineers and managers as part of the DARE hyper-platform as they create, validate and use their data-driven scientific workflows.
- 1 April 2022
- Published by China Science Publishing & Media Ltd. in Data Intelligence
- Vol. 4 (2) , 226-242
Open AccessS-ProvFlow. Storing and Exploring Lineage Data as a Service
We present a set of configurable Web service and interactive tools, s-ProvFlow, for managing and exploiting records tracking data lineage during workflow runs. It facilitates detailed analysis of single executions. It helps users manage complex tasks by exposing the relationships between data, people, equipment and workflow runs intended to combine productively. Its logical model extends the PROV standard to precisely record parallel data-streaming applications. Its metadata handling encourages users to capture the application context by specifying how application attributes, often using standard vocabularies, should be added. These metadata records immediately help productivity as the interactive tools support their use in selection and bulk operations. Users rapidly appreciate the power of the encoded semantics as they reap the benefits. This improves the quality of provenance for users and management. Which in turn facilitates analysis of collections of runs, enabling users to manage results and validate procedures. It fosters reuse of data and methods and facilitates diagnostic investigations and optimisations. We present S-ProvFlow's use by scientists, research engineers and managers as part of the DARE hyper-platform as they create, validate and use their data-driven scientific workflows.
- Open Access
- 27 March 2022
- Published by Copernicus GmbH
Better Tailoring of Climate Information for End Users using Targeted Interfaces and Tools
End users of climate change information are relying on climate services and tools in order to produce meaningful information for specific applications. Data volumes as well as the number of datasets are increasing very rapidly, and the ability to select, process and download all needed data is getting complex, technical and very time-consuming. It is especially true since those datasets are often distributed among several data centres and into a large quantity of files.Several platform are being developed to hide this complexity to users and provide a seamless access to climate data, as well as to provide on-demand data analysis capabilities. We can cite, for example, the Copernicus Data Store (CDS https://cds.climate.copernicus.eu), along with its toolbox to perform online data analysis. Another platform is developed within the H2020 IS-ENES3 project, called climate4impact (C4I 2.0 https://dev.climate4impact.eu ). It is using an enhanced Jupyter-Lab environment called SWIRRL (Software for Interactive Reproducible Research Labs https://gitlab.com/KNMI-OSS/swirrl ) along with a collection of Jupyter notebooks (https://gitlab.com/is-enes-cdi-c4i/notebooks) as useful set of example on how to use the data.Finally, the portal provides interactive pages for the evaluation of climate models (using ESMValTool) to guide users on selecting climate datasets. The notebooks that can be executed in C4I, are developed using a very convenient software library, which is made available via SWIRRL, to calculate climate indices and indicators called icclim (v5.0 https://github.com/cerfacs-globc/icclim ). This library, which is also in the process of being integrated into the C3S, is a flexible python software package to calculate climate indices and indicators. This tool adhere as much as possible to metadata conventions such as the Climate & Forecasting Conventions (CF-1.x) as well as the clix-meta (https://github.com/clix-meta) work that is being done in IS-ENES3. Proper provenance information still needs to be added. The ultimate goal is to be as close as possible to all FAIR aspects. icclim is designed with performance and optimisation in mind, because the goal is to provide on-demand calculations for users. It provides the implementation of most of the international standard climate indices such as ECAD, ETCCDI, ET-SCI, including the correct methodology for calculating percentile indices using the bootstrapping method. It has been validated against R.Climdex as well (https://cran.r-project.org/web/packages/climdex.pcic/index.html). This new 5.x version of icclim is based on functions from the xclim (https://github.com/Ouranosinc/xclim) python library, which was inspired by earlier versions of icclim, but using xarray and dask for data access and processing.In this presentation, the climate4impact 2.0 platform will be described along with the icclim climate indices tool. Important metadata aspects will also be discussed (clix-meta). A few examples using the jupyter notebook collection will be shown. This project (IS-ENES3) has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N°824084.
- 27 March 2022
- Published by Copernicus GmbH
Open AccessBetter Tailoring of Climate Information for End Users using Targeted Interfaces and Tools
End users of climate change information are relying on climate services and tools in order to produce meaningful information for specific applications. Data volumes as well as the number of datasets are increasing very rapidly, and the ability to select, process and download all needed data is getting complex, technical and very time-consuming. It is especially true since those datasets are often distributed among several data centres and into a large quantity of files.Several platform are being developed to hide this complexity to users and provide a seamless access to climate data, as well as to provide on-demand data analysis capabilities. We can cite, for example, the Copernicus Data Store (CDS https://cds.climate.copernicus.eu), along with its toolbox to perform online data analysis. Another platform is developed within the H2020 IS-ENES3 project, called climate4impact (C4I 2.0 https://dev.climate4impact.eu ). It is using an enhanced Jupyter-Lab environment called SWIRRL (Software for Interactive Reproducible Research Labs https://gitlab.com/KNMI-OSS/swirrl ) along with a collection of Jupyter notebooks (https://gitlab.com/is-enes-cdi-c4i/notebooks) as useful set of example on how to use the data.Finally, the portal provides interactive pages for the evaluation of climate models (using ESMValTool) to guide users on selecting climate datasets. The notebooks that can be executed in C4I, are developed using a very convenient software library, which is made available via SWIRRL, to calculate climate indices and indicators called icclim (v5.0 https://github.com/cerfacs-globc/icclim ). This library, which is also in the process of being integrated into the C3S, is a flexible python software package to calculate climate indices and indicators. This tool adhere as much as possible to metadata conventions such as the Climate & Forecasting Conventions (CF-1.x) as well as the clix-meta (https://github.com/clix-meta) work that is being done in IS-ENES3. Proper provenance information still needs to be added. The ultimate goal is to be as close as possible to all FAIR aspects. icclim is designed with performance and optimisation in mind, because the goal is to provide on-demand calculations for users. It provides the implementation of most of the international standard climate indices such as ECAD, ETCCDI, ET-SCI, including the correct methodology for calculating percentile indices using the bootstrapping method. It has been validated against R.Climdex as well (https://cran.r-project.org/web/packages/climdex.pcic/index.html). This new 5.x version of icclim is based on functions from the xclim (https://github.com/Ouranosinc/xclim) python library, which was inspired by earlier versions of icclim, but using xarray and dask for data access and processing.In this presentation, the climate4impact 2.0 platform will be described along with the icclim climate indices tool. Important metadata aspects will also be discussed (clix-meta). A few examples using the jupyter notebook collection will be shown. This project (IS-ENES3) has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N°824084.