Evaluating the impact of task aggregation in workflows with shared resource environments: use case for the MONARCH application
Open Access
- 5 December 2025
- journal article
- Published by Copernicus GmbH in Geoscientific Model Development
- Vol. 18 (23) , 9709-9721
Abstract
High Performance Computing (HPC) is commonly employed to run high-impact Earth System Model (ESM) simulations, such as those for climate change. However, running workflows of ESM simulations on cutting-edge platforms can take a long time due to the congestion of the system and the lack of coordination between current HPC schedulers and workflow manager systems (WfMS). The Earth Sciences community has estimated the time in queue to be between 10 % to 20 % of the runtime in climate prediction experiments, the most time-consuming exercise. To address this issue, the developers of Autosubmit, a WfMS tailored for climate and air quality sciences, have developed wrappers to submit multiple subsequent workflow tasks – the indivisible unit of compute as defined by the workflow – in a single remote job, without altering any of the tasks. However, although wrappers are widely used in production for community models such as EC-Earth3, MONARCH, and Destination Earth simulations, to our knowledge, the benefits and potential drawbacks have never been rigorously evaluated. Later, the developers of Autosubmit noticed that the impact of using wrappers was related to the past utilization of the user and its group, which the popular scheduler Slurm uses to compute the priority of the queueing jobs. In Slurm's methodology, this past utilization is quantified in the fair share factor. The objective of this paper is to quantify the impact of wrapping subsequent tasks on queue time and understand its relationship with the fair share and the job's CPU and runtime request. To do this, we used a Slurm simulator to reproduce the behavior of the scheduler and, to recreate a representative usage of an HPC platform, we generated synthetic static workloads from data of the LUMI supercomputer and a dynamic workload from a past flagship HPC platform. As an example, we introduced jobs modeled after the MONARCH air quality application in these workloads, and we tracked their queue time. We found that, by simply joining tasks, the total time-to-solution of the simulation reduces up to 7 % with respect to the runtime of the simulations, and we believe that this value is larger the longer the workflow, since longer wrappers could be created and hence less jobs are submitted to the scheduler. This saving translates to absolute terms of about eight days less wasted in queue time for half of the simulations from the IS-ENES3 consortium of CMIP6 simulations. We also identified in the static experiments a high inverse correlation of −0.87, between the queue time and the fair share factor.Keywords
All Related Versions
Funding Information
- Ministerio de Ciencia e Innovación (CEX2021-001148-S-20-5)
- Consejo Superior de Investigaciones Científicas (PID2020-116324RA-I00)
This publication has 21 references indexed in Scilit:
- Architecture of the Slurm Workload Manager2023
- Destination Earth – A digital twin in support of climate servicesClimate Services, 2023
- Design and Performance Characterization of RADICAL-Pilot on Leadership-Class PlatformsIEEE Transactions on Parallel and Distributed Systems, 2021
- Job Characteristics on Large-Scale Systems: Long-Term Analysis, Quantification, and Implications2020
- AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenanceScientific Data, 2020
- Evaluating SLURM Simulator with Real-Machine SLURM and Vice Versa2018
- Seamless management of ensemble climate prediction experiments on HPC platforms2016
- Scheduling Algorithms2007
- Selective Reservation Strategies for Backfill Job Scheduling2002
- Adaptive Selection of Partition Size for Supercomputer Requests2000