Accelerating Function-Centric Applications by Discovering, Distributing, and Retaining Reusable Context in Workflow Systems
Open Access
- 3 June 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 122-134
Abstract
Workflow systems provide a convenient way for users to write large-scale applications by composing independent tasks into large graphs that can be executed concurrently on high-performance clusters. In many newer workflow systems, tasks are often expressed as a combination of function invocations in a high-level language. Because necessary code and data are not statically known prior to execution, they must be moved into the cluster at runtime. An obvious way of doing this is to translate function invocations into self-contained executable programs and run them as usual, but this brings a hefty performance penalty: a function invocation now needs to piggyback its context with extra code and data to a remote node, and the remote node needs to take extra time to reconstruct the invocation's context before executing it, both detrimental to lightweight short-running functions. A better solution for workflow systems is to treat functions and invocations as first-class abstractions: subsequent invocations of the same function on a worker node should only pay for the cost of context setup once and reuse the context between different invocations. The remaining problems lie in discovering, distributing, and retaining the reusable context among workers. In this paper, we discuss the rationale and design requirement of these mechanisms to support context reuse, and implement them in TaskVine, a data-intensive distributed framework and execution engine. Our results from executing a large-scale neural network inference application and a molecular design application show that treating functions and invocations as first-class abstractions reduces the execution time of the applications by 94.5% and 26.9%, respectively.Keywords
Funding Information
- NSF (National Science Foundation) (OAC-1931348)
This publication has 19 references indexed in Scilit:
- Landlord: Coordinating Dynamic Software Environments to Reduce Container SprawlIEEE Transactions on Parallel and Distributed Systems, 2023
- Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing2021
- Quantum Chemistry-Informed Active Learning to Accelerate the Design and Discovery of Sustainable Energy Storage MaterialsChemistry of Materials, 2020
- Middleware Building Blocks for Workflow SystemsComputing in Science & Engineering, 2019
- GekkoFS - A Temporary Distributed File System for HPC Applications2018
- Deploying High Throughput Scientific Workflows on Container Schedulers with Makeflow and Mesos2017
- An Ephemeral Burst-Buffer File System for Scientific Applications2016
- Deep Residual Learning for Image Recognition2016
- Pegasus, a workflow management system for science automationFuture Generation Computer Systems, 2015
- Optimization of parameters for semiempirical methods VI: more modifications to the NDDO approximations and re-optimization of parametersJournal of Molecular Modeling, 2012