Design and Performance Characterization of RADICAL-Pilot on Leadership-Class Platforms
- 19 August 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (4) , 818-829
Abstract
Many extreme scale scientific applications have workloads comprised of a large number of individual high-performance tasks. The Pilot abstraction decouples workload specification, resource management, and task execution via job placeholders and late-binding. As such, suitable implementations of the Pilot abstraction can support the collective execution of large number of tasks on supercomputers. We introduce RADICAL-Pilot (RP) as a portable, modular and extensible pilot-enabled runtime system. We describe RP's design, architecture and implementation. We characterize its performance and show its ability to scalably execute workloads comprised of tens of thousands heterogeneous tasks on DOE and NSF leadership-class HPC platforms. Specifically, we investigate RP's weak/strong scaling with CPU/GPU, single/multi core, (non)MPI tasks and Python functions when using most of ORNL Summit and TACC Frontera. RADICAL-Pilot can be used stand-alone, as well as the runtime for third-party workflow systems.Keywords
All Related Versions
Funding Information
- National Science Foundation (NSF-1931512, NSF-1835449)
- ECP CANDLE and ExaWorks
- Exascale Computing Project 17-SC-20-SC (DESC0012704)
- U.S. Department of Energy (DE-AC05-00OR22725)
- XSEDE resources (TG-MCB090174)
This publication has 35 references indexed in Scilit:
- GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobsFuture Generation Computer Systems, 2015
- Evolution of the ATLAS PanDA workload management system for exascale computational scienceJournal of Physics: Conference Series, 2014
- Fast recovery of free energy landscapes via diffusion-map-directed molecular dynamicsPhysical Chemistry Chemical Physics, 2014
- JETS: Language and System Support for Many-Parallel-Task WorkflowsJournal of Grid Computing, 2013
- Eden: Simplified Management of Atypical High-Performance Computing JobsComputing in Science & Engineering, 2012
- DIRAC pilot framework and the DIRAC Workload Management SystemJournal of Physics: Conference Series, 2010
- Pilot factory – a Condor-based system for scalable Pilot Job generation in the Panda WMS frameworkJournal of Physics: Conference Series, 2010
- glideinWMS—a generic pilot-based workload management systemJournal of Physics: Conference Series, 2008
- The open science gridJournal of Physics: Conference Series, 2007
- CilkACM SIGPLAN Notices, 1995