Empowering Agroecosystem Modeling with HTC Scientific Workflows: The Cycles Model Use Case
- 1 December 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4545-4552
Abstract
Scientific workflows have enabled large-scale scientific computations and data analysis, and lowered the entry barrier for performing computations in distributed heterogeneous platforms (e.g., HTC and HPC). In spite of impressive achievements to date, large-scale modeling, simulation, and data analytics in the long-tail still face several challenges such as efficient scheduling and execution of large-scale workflows $(\mathrm{O}(10^{6}))$ with very short-running tasks (few seconds). While the current trend to support next-generation workflows on leadership class machines have gained much attention in the past years, at the other end of the spectrum scientific workflows from the long-tail science have become larger and require processing massive volumes of data. In this paper, we report on our experience in designing and implementing an HTC workflow for agroecosystem modeling. We leverage well-known (task clustering and co-scheduling) and emerging (hierarchical workflows and containers) workflow optimization techniques to make the workflow planning problem tractable, and maximize resource utilization and the degree of task parallelism. Experimental results, via the implementation of a use case, show that by strategically combining the above strategies and defining an appropriate set of optimization parameters, the overall workflow makespan can be improved by 3.5 orders of magnitude when compared to a regular (non-optimized) execution of the workflow.
Keywords
This publication has 28 references indexed in Scilit:
- Dynamic steering of HPC scientific workflows: A surveyFuture Generation Computer Systems, 2015
- Pegasus, a workflow management system for science automationFuture Generation Computer Systems, 2015
- Implications of carbon saturation model structures for simulated nitrogen mineralization dynamicsBiogeosciences, 2014
- CropSyst model evolution: From field to regional to global scales and from research to decision support systemsEnvironmental Modelling & Software, 2014
- Automating data-model workflows at a level 12 HUC scale: Watershed modeling in a distributed computing environmentEnvironmental Modelling & Software, 2014
- C-Farm: A simple model to evaluate the carbon balance of soil profilesEuropean Journal of Agronomy, 2010
- A simple method to estimate harvest index in grain cropsField Crops Research, 2007
- A recommendation on standardized surface resistance for hourly calculation of reference ETo by the FAO56 Penman-Monteith methodAgricultural Water Management, 2006
- Distributed computing in practice: the Condor experienceConcurrency and Computation: Practice and Experience, 2005
- Simulation of Water Uptake in Maize, Using Different Levels of Process DetailAgronomy Journal, 1999