Evaluating the Impact of Task Aggregation in Workflows with Shared Resource Environments: use case for the MONARCH application
Preprint
- 20 May 2025
- preprint
- Published by Copernicus GmbH in EGUsphere
- Vol. 2025, 1-18
Abstract
High Performance Computing (HPC) is commonly employed to run high-impact Earth System Model (ESM) simulations, such as those for climate change. However, running workflows of ESM simulations on cutting-edge platforms can take long due to the congestion of the system and the lack of coordination between current HPC schedulers and workflow manager systems (WfMS). The Earth Sciences community has estimated the time in queue to be between 10 % to 20 % of the runtime in climate prediction experiments, the most time-consuming exercise. To address this issue, the developers of Autosubmit, a WfMS tailored for climate and air quality sciences, have developed wrappers to join multiple subsequent workflow tasks into a single submission. However, although wrappers are widely used in production for community models such as EC-Earth3, MONARCH, and Destination Earth simulations, to our knowledge, the benefits and potential drawbacks have never been rigorously evaluated. In addition, with portability in mind, the developers proposed to wrap depending on the entitlement of the user to the machine. In the widely utilized Slurm scheduler, this factor is called fair share. The objective of this paper is to quantify the impact of wrapping on queue time and understand its relationship with the fair share and the job's CPU and runtime request. To do this, we used a Slurm simulator to reproduce the behavior of the scheduler and, to recreate a representative usage of an HPC platform, we generated synthetic static workloads from data of the LUMI supercomputer and a dynamic workload from a past flagship HPC platform. As an example, we introduced jobs modeled after the MONARCH air quality application in these workloads, which we tracked their queue time. We found that, by simply joining tasks, the total runtime of the simulation reduces up to 7 %, and we have indications that this value is larger in reality. This saving translates to absolute terms in at least eight days less wasted in queue time for half of the simulations from the IS-ENES3 consortium of CMIP6 simulations. We also identified a high inverse correlation, of -0.87, between the queue time and the fair share factor.Keywords
All Related Versions
- Published version: Geoscientific Model Development, 18 (23), 9709.
Funding Information
- Ministerio de Ciencia e Innovación (CEX2021-001148-S-20-5)
- Consejo Superior de Investigaciones Científicas (PID2020-116324RA-I00)
This publication has 7 references indexed in Scilit:
- The computational and energy cost of simulation and storage for climate science: lessons from CMIP6Geoscientific Model Development, 2024
- Destination Earth – A digital twin in support of climate servicesClimate Services, 2023
- Job Scheduling Strategies for Parallel Processing2023
- Novel Proposals for FAIR, Automated, Recommendable, and Robust Workflows2022
- The EC-Earth3 Earth system model for the Coupled Model Intercomparison Project 6Geoscientific Model Development, 2022
- Improving ocean modeling software NEMO 4.0 benchmarking and communication efficiencyGeoscientific Model Development, 2022
- Seamless management of ensemble climate prediction experiments on HPC platforms2016