PGPablo Goitia

Pablo Goitia

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Pablo Goitia has a total of 3 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Task aggregation as a strategy to optimize Earth System Model workflows in HPC: assessing real scenarios with EC-Earth

    by Pablo Goitia, Manuel G. Marciani, Miguel Castrillo, Mario C. Acosta

    Earth System Models (ESMs) are commonly executed as complex workflows consisting of numerous interdependent tasks that comprise steps such as model and data deployment, simulation, data transfer, and post-processing. Workflows facilitate the execution of long high-resolution configurations by splitting the runtime of the simulation into different tasks, in order to ensure frequent checkpointing and to comply with the restrictions of the large High-Performance Computing (HPC) machines where they are executed. These machines are frequently congested due to their high demand and, therefore, implement scheduling policies to share the resources among the users. This complicates the execution of long ensemble simulation workflows, where each successive simulation task can only be submitted once the preceding one finishes, causing queue time to accumulate, which is the duration for which the jobs wait for the HPC platform to allocate the required resources for their execution. To alleviate this issue, we propose achieving shorter times-to-response, which are the intervals from the first submission to the completion of the final task, by applying a solution to reduce subsequent requests for resources and, consequently, reducing queue times. This solution, known as task aggregation, consists of grouping multiple tasks and submitting them as a single job, respecting their dependencies, and without altering their underlying logic. This technique separates the workflow definition from how individual tasks are submitted and executed, allowing the workflow manager to make decisions during runtime. In this paper, we performed the first controlled assessment on the effects of task aggregation by conducting concurrent pairs of climate simulations in production machines, with the sole difference that one uses aggregation. These simulations were executed on three European supercomputers: MeluXina, MareNostrum 4, and MareNostrum 5. We measured the evolution of the fair share, a scheduling factor that normally plays a major role in the priority of the jobs. Besides absolute gains, we compute the impact of aggregation by comparing consolidated performance metrics for climate models within the community. We prove the benefits of the task aggregation using EC-Earth3, a widely used European community climate model that shares main features with many other ESMs and has a representative workload. The experimental findings of our research indicate that across the three evaluated supercomputing platforms, applying task aggregation decreases total queue times by 11.17 to 12.33 times compared to a workflow that does not , representing an improvement of up to 23 % more years simulated in the same time span in the case of the platform with the highest congestion. Therefore, results have shown that task aggregation proves to be beneficial for long climate simulations. Moreover, we have credible reasons to believe that any vertical (also called chained) workflow should benefit from using it. We explain that this reduction in the time-to-solution comes from the decrease in the number of submitted jobs and the utilization of the machine. By aggregating tasks, we had many times less jobs queued and, albeit their longer length, we observed that they are in queue less time than if they had been submitted individually. Our results also show that the user's jobs are being held in queue in spite of their utilization due to the fair share being influenced by the other members of the HPC account, which is a direct consequence of the fair share policy of the machine.

  • Open Access

    Reply on RC2

    by Pablo Goitia, Manuel G. Marciani, Miguel Castrillo, Mario C. Acosta

    Earth System Models (ESMs) are commonly executed as complex workflows consisting of numerous interdependent tasks – the atomic unit of computation within the workflow – which comprise steps as model and data deployment, simulation, data transfer, and post-processing. Workflows facilitate the execution of long high-resolution configurations by splitting the runtime of the simulation into different tasks, in order to ensure frequent checkpointing and to comply with the restrictions of the large High-Performance Computing (HPC) machines where they are executed. These machines are frequently congested and, therefore, implement scheduling policies to share the resources among the users, complicating the execution of long ensemble simulation workflows due to the accumulated queue time, because each successive simulation task can only be submitted once the preceding one finishes. These queue times are the duration for which the jobs – the compute units sent to be executed remotely – wait for the HPC platform to allocate the required resources for their execution. To alleviate this issue, we propose achieving shorter times-to-response, which are the durations from the first submission to the completion of the final task, by applying task aggregation to reduce subsequent requests for resources and, consequently, reducing queue times. Task aggregation is a strategy that consists of grouping multiple tasks and submitting them as a single job, respecting their dependencies, and without altering their underlying logic. In this paper, we performed the first controlled assessment on the effects of task aggregation by conducting concurrent pairs of climate simulations in production machines, with the sole difference that one uses aggregation. These simulations were executed on three European supercomputers: MeluXina, MareNostrum 4, and MareNostrum 5. We measured the evolution of the fair share, a scheduling factor that normally plays a major role in the priority of the jobs. Besides absolute time, we compute the impact of aggregation by obtaining the differences between the Simulated Years Per Day (SYPD) and the Actual Simulated Years Per Day (ASYPD), two consolidated performance metrics for climate models within the community. We prove the benefits of the task aggregation using EC-Earth3, a widely used European community climate model that shares main features with many other ESMs and has a representative workload. The experimental findings of our research indicate that across the three evaluated supercomputing platforms, applying task aggregation decreases total queue times by 11.17 to 12.33 times compared to a workflow that does not, representing an improvement in the ASYPD of up to 23,04 % in the case of the platform with the highest congestion. Therefore, results have shown that task aggregation proves to be beneficial for long climate simulations. Moreover, we have credible reasons to believe that any vertical (also called chained) workflow should benefit from using it. We explain that this reduction in the time-to-solution comes from the decrease in the number of submitted jobs and the congestion of the machine. By aggregating tasks, we had many times less jobs queued and, albeit their longer length, we observed that they are in queue less time than if they had been submitted individually. Our results also show that the user's jobs are being held in queue in spite of their utilization due to the fair share being influenced by the other members of the HPC account, which is a direct consequence of the fair share policy of the machine.

  • Open Access

    Comment on egusphere-2026-539

    by Pablo Goitia, Manuel G. Marciani, Miguel Castrillo, Mario C. Acosta

    Earth System Models (ESMs) are commonly executed as complex workflows consisting of numerous interdependent tasks – the atomic unit of computation within the workflow – which comprise steps as model and data deployment, simulation, data transfer, and post-processing. Workflows facilitate the execution of long high-resolution configurations by splitting the runtime of the simulation into different tasks, in order to ensure frequent checkpointing and to comply with the restrictions of the large High-Performance Computing (HPC) machines where they are executed. These machines are frequently congested and, therefore, implement scheduling policies to share the resources among the users, complicating the execution of long ensemble simulation workflows due to the accumulated queue time, because each successive simulation task can only be submitted once the preceding one finishes. These queue times are the duration for which the jobs – the compute units sent to be executed remotely – wait for the HPC platform to allocate the required resources for their execution. To alleviate this issue, we propose achieving shorter times-to-response, which are the durations from the first submission to the completion of the final task, by applying task aggregation to reduce subsequent requests for resources and, consequently, reducing queue times. Task aggregation is a strategy that consists of grouping multiple tasks and submitting them as a single job, respecting their dependencies, and without altering their underlying logic. In this paper, we performed the first controlled assessment on the effects of task aggregation by conducting concurrent pairs of climate simulations in production machines, with the sole difference that one uses aggregation. These simulations were executed on three European supercomputers: MeluXina, MareNostrum 4, and MareNostrum 5. We measured the evolution of the fair share, a scheduling factor that normally plays a major role in the priority of the jobs. Besides absolute time, we compute the impact of aggregation by obtaining the differences between the Simulated Years Per Day (SYPD) and the Actual Simulated Years Per Day (ASYPD), two consolidated performance metrics for climate models within the community. We prove the benefits of the task aggregation using EC-Earth3, a widely used European community climate model that shares main features with many other ESMs and has a representative workload. The experimental findings of our research indicate that across the three evaluated supercomputing platforms, applying task aggregation decreases total queue times by 11.17 to 12.33 times compared to a workflow that does not, representing an improvement in the ASYPD of up to 23,04 % in the case of the platform with the highest congestion. Therefore, results have shown that task aggregation proves to be beneficial for long climate simulations. Moreover, we have credible reasons to believe that any vertical (also called chained) workflow should benefit from using it. We explain that this reduction in the time-to-solution comes from the decrease in the number of submitted jobs and the congestion of the machine. By aggregating tasks, we had many times less jobs queued and, albeit their longer length, we observed that they are in queue less time than if they had been submitted individually. Our results also show that the user's jobs are being held in queue in spite of their utilization due to the fair share being influenced by the other members of the HPC account, which is a direct consequence of the fair share policy of the machine.

  • Open Access

    Reply on RC1

    by Pablo Goitia

  • Open Access

    Comment on egusphere-2026-539

    by Pablo Goitia, Manuel G. Marciani, Miguel Castrillo, Mario C. Acosta

    Earth System Models (ESMs) are commonly executed as complex workflows consisting of numerous interdependent tasks – the atomic unit of computation within the workflow – which comprise steps as model and data deployment, simulation, data transfer, and post-processing. Workflows facilitate the execution of long high-resolution configurations by splitting the runtime of the simulation into different tasks, in order to ensure frequent checkpointing and to comply with the restrictions of the large High-Performance Computing (HPC) machines where they are executed. These machines are frequently congested and, therefore, implement scheduling policies to share the resources among the users, complicating the execution of long ensemble simulation workflows due to the accumulated queue time, because each successive simulation task can only be submitted once the preceding one finishes. These queue times are the duration for which the jobs – the compute units sent to be executed remotely – wait for the HPC platform to allocate the required resources for their execution. To alleviate this issue, we propose achieving shorter times-to-response, which are the durations from the first submission to the completion of the final task, by applying task aggregation to reduce subsequent requests for resources and, consequently, reducing queue times. Task aggregation is a strategy that consists of grouping multiple tasks and submitting them as a single job, respecting their dependencies, and without altering their underlying logic. In this paper, we performed the first controlled assessment on the effects of task aggregation by conducting concurrent pairs of climate simulations in production machines, with the sole difference that one uses aggregation. These simulations were executed on three European supercomputers: MeluXina, MareNostrum 4, and MareNostrum 5. We measured the evolution of the fair share, a scheduling factor that normally plays a major role in the priority of the jobs. Besides absolute time, we compute the impact of aggregation by obtaining the differences between the Simulated Years Per Day (SYPD) and the Actual Simulated Years Per Day (ASYPD), two consolidated performance metrics for climate models within the community. We prove the benefits of the task aggregation using EC-Earth3, a widely used European community climate model that shares main features with many other ESMs and has a representative workload. The experimental findings of our research indicate that across the three evaluated supercomputing platforms, applying task aggregation decreases total queue times by 11.17 to 12.33 times compared to a workflow that does not, representing an improvement in the ASYPD of up to 23,04 % in the case of the platform with the highest congestion. Therefore, results have shown that task aggregation proves to be beneficial for long climate simulations. Moreover, we have credible reasons to believe that any vertical (also called chained) workflow should benefit from using it. We explain that this reduction in the time-to-solution comes from the decrease in the number of submitted jobs and the congestion of the machine. By aggregating tasks, we had many times less jobs queued and, albeit their longer length, we observed that they are in queue less time than if they had been submitted individually. Our results also show that the user's jobs are being held in queue in spite of their utilization due to the fair share being influenced by the other members of the HPC account, which is a direct consequence of the fair share policy of the machine.

  • Open Access

    Task aggregation as a strategy to optimize Earth System Model workflows in HPC: assessing real scenarios with EC-Earth

    by Pablo Goitia, Manuel G. Marciani, Miguel Castrillo, Mario C. Acosta

    Earth System Models (ESMs) are commonly executed as complex workflows consisting of numerous interdependent tasks – the atomic unit of computation within the workflow – which comprise steps as model and data deployment, simulation, data transfer, and post-processing. Workflows facilitate the execution of long high-resolution configurations by splitting the runtime of the simulation into different tasks, in order to ensure frequent checkpointing and to comply with the restrictions of the large High-Performance Computing (HPC) machines where they are executed. These machines are frequently congested and, therefore, implement scheduling policies to share the resources among the users, complicating the execution of long ensemble simulation workflows due to the accumulated queue time, because each successive simulation task can only be submitted once the preceding one finishes. These queue times are the duration for which the jobs – the compute units sent to be executed remotely – wait for the HPC platform to allocate the required resources for their execution. To alleviate this issue, we propose achieving shorter times-to-response, which are the durations from the first submission to the completion of the final task, by applying task aggregation to reduce subsequent requests for resources and, consequently, reducing queue times. Task aggregation is a strategy that consists of grouping multiple tasks and submitting them as a single job, respecting their dependencies, and without altering their underlying logic. In this paper, we performed the first controlled assessment on the effects of task aggregation by conducting concurrent pairs of climate simulations in production machines, with the sole difference that one uses aggregation. These simulations were executed on three European supercomputers: MeluXina, MareNostrum 4, and MareNostrum 5. We measured the evolution of the fair share, a scheduling factor that normally plays a major role in the priority of the jobs. Besides absolute time, we compute the impact of aggregation by obtaining the differences between the Simulated Years Per Day (SYPD) and the Actual Simulated Years Per Day (ASYPD), two consolidated performance metrics for climate models within the community. We prove the benefits of the task aggregation using EC-Earth3, a widely used European community climate model that shares main features with many other ESMs and has a representative workload. The experimental findings of our research indicate that across the three evaluated supercomputing platforms, applying task aggregation decreases total queue times by 11.17 to 12.33 times compared to a workflow that does not, representing an improvement in the ASYPD of up to 23,04 % in the case of the platform with the highest congestion. Therefore, results have shown that task aggregation proves to be beneficial for long climate simulations. Moreover, we have credible reasons to believe that any vertical (also called chained) workflow should benefit from using it. We explain that this reduction in the time-to-solution comes from the decrease in the number of submitted jobs and the congestion of the machine. By aggregating tasks, we had many times less jobs queued and, albeit their longer length, we observed that they are in queue less time than if they had been submitted individually. Our results also show that the user's jobs are being held in queue in spite of their utilization due to the fair share being influenced by the other members of the HPC account, which is a direct consequence of the fair share policy of the machine.

  • Open Access

    Optimizing the Destination Earth Workflow with in situ HPC Task Orchestration

    by Pablo Goitia, Manuel Giménez de Castro Marciani, Miguel Castrillo

    Traditionally, climate simulations are executed on High-Performance Computing (HPC) platforms, organized in workflows that involve all the steps for the complete execution of the model, data processing, and management tasks. With the sustained increase in the computing capacity of these machines over the years, the accuracy and resolution of climate simulations have reached levels never seen before.In this context, the European Commission launched the Destination Earth initiative, aimed at developing a digital twin of the Earth for the adaptation to climate change. This initiative seeks to operationalize the running of very high-resolution climate simulations that are coupled with applications that consume their data as it is produced. In order to address the challenge of processing the hundreds of terabytes that each single simulation involves, the ClimateDT project implemented a data streaming approach. This means that any delay between the production time of the climate model data and the subsequent consumption by the post-processing applications results in a workflow misalignment, leading to unacceptable delays in the total execution time. This poses unprecedented challenges on the workflow management side.One of the main causes of the misalignments that commonly occur lies in the long time that each of the many thousands of tasks of the workflow spends in the queues of the HPC job schedulers, such as Slurm. To address this issue, the community proposed to aggregate workflow tasks into a single submission to the HPC without altering their execution logic—a technique known as task aggregation. Previous studies have demonstrated the effectiveness of this approach for climate workflows, yielding promising results. However, the current implementation is limited, as the task execution within an allocation still relies on the workflow manager, which is not able to perform the fine-grained workflow orchestration that a dedicated tool could do in a convenient way.To overcome this limitation, we propose in this work to integrate existing HPC software into the Autosubmit Workflow Manager to enable in situ orchestration of aggregated tasks, such as the renowned Flux Framework and Parsl. This integration aims to abstract both developers and users from the complexity of managing supercomputing resources, providing an easy-to-use interface. The proposed approach is validated using the Destination Earth workflow to enable more complex, structured forms of task aggregation while reducing queue times in large-scale simulations.