End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Current scientific workflow systems do not typically integrate simulation-centric and data-centric aspects due to their very different software/infrastructure requirements. A transparent integration of such components into a single end-to-end workflow would lead to a more efficient and automated way for generating insights from large simulation data. This work presents a complex case study related to extreme events analysis of future climate data that integrates in the same workflow numerical simulations, Big Data analytics and Machine Learning models. The case study is being implemented in the context of the eFlows4HPC project using the project’s software stack for deployment and orchestration of the workflow. The solution implemented in the project has shown to simplify the development and execution of end-to-end climate workflows with heterogeneous software requirements. Moreover, such an approach can, in the long term, increase the reuse of workflows by scientists and their portability over different HPC infrastructures.
Funding Information
  • Generalitat de Catalunya (contract 2021-SGR-00412)
  • H2020 Industrial Leadership (955558)
  • European Union NextGenerationEU/PRTR (PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021-121927)
  • Spanish Government (contract PID2019-107255 GB)
  • MCIN/AEI/10.13039/501100011033
  • Severo Ochoa Programme for Centres of Excellence in R&D (CEX2021-001148-S)

This publication has 37 references indexed in Scilit: