A Comprehensive Perspective on Pilot-Job Systems
Authors:
- Mark SantcroosMSMark Santcroos
- Shantenu JhaSJShantenu Jha
Open Access
- 17 April 2018
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 51 (2) , 1-32
Abstract
Pilot-Job systems play an important role in supporting distributed scientific computing. They are used to execute millions of jobs on several cyberinfrastructures worldwide, consuming billions of CPU hours a year. With the increasing importance of task-level parallelism in high-performance computing, Pilot-Job systems are also witnessing an adoption beyond traditional domains. Notwithstanding the growing impact on scientific research, there is no agreement on a definition of Pilot-Job system and no clear understanding of the underlying abstraction and paradigm. Pilot-Job implementations have proliferated with no shared best practices or open interfaces and little interoperability. Ultimately, this is hindering the realization of the full impact of Pilot-Jobs by limiting their robustness, portability, and maintainability. This article offers a comprehensive analysis of Pilot-Job systems critically assessing their motivations, evolution, properties, and implementation. The three main contributions of this article are as follows: (1) an analysis of the motivations and evolution of Pilot-Job systems; (2) an outline of the Pilot abstraction, its distinguishing logical components and functionalities, its terminology, and its architecture pattern; and (3) the description of core and auxiliary properties of Pilot-Jobs systems and the analysis of six exemplar Pilot-Job implementations. Together, these contributions illustrate the Pilot paradigm, its generality, and how it helps to address some challenges in distributed scientific computing.Keywords
All Related Versions
Funding Information
- NSF CAREER (ACI-1253644 and ACI-1440677)
- Department of Energy Award (DE- FG02-12ER26115)
This publication has 83 references indexed in Scilit:
- Campus Grids: Bringing Additional Computational Resources to HEP ResearchersJournal of Physics: Conference Series, 2012
- Multi-core job submission and grid resource scheduling for ATLAS AthenaMPJournal of Physics: Conference Series, 2012
- Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHCPhysics Letters B, 2012
- Swift: A language for distributed parallel scriptingParallel Computing, 2011
- DIRAC pilot framework and the DIRAC Workload Management SystemJournal of Physics: Conference Series, 2010
- Ganga: A tool for computational-task management and easy access to Grid resourcesComputer Physics Communications, 2009
- Flexible and Efficient Workflow Deployment of Data-Intensive Applications On Grids With MOTEURThe International Journal of High Performance Computing Applications, 2008
- The ALICE experiment at the CERN LHCJournal of Instrumentation, 2008
- Multi-level scheduling for global optimization in grid computingComputers and Electrical Engineering, 2008
- The abstract machine and implementation of parallel ParlogNew Generation Computing, 1992