Colin Thomas
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Colin Thomas has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Douglas Thain
University of Notre Dame
2 shared publications - Thanh Son Phung
University of Notre Dame
2 shared publications - Andrew Hennessee
University of Notre Dame, South Bend, Indiana, USA
1 shared publication - Barry Sly-Delgado
University of Notre Dame, United States of America
1 shared publication - Benjamin Tovar
Department of Computer Science and Engineering, University of Notre Dame, 6111 Notre Dame, Indiana United States
1 shared publication C.A. Kent
SERC Centre for Biochemical Engineering, School of Chemical Engineering, University of Birmingham, Birmingham, B15 2TT, UK
1 shared publication
Latest Publications
- Open Access
- 3 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 122-134
Accelerating Function-Centric Applications by Discovering, Distributing, and Retaining Reusable Context in Workflow Systems
Workflow systems provide a convenient way for users to write large-scale applications by composing independent tasks into large graphs that can be executed concurrently on high-performance clusters. In many newer workflow systems, tasks are often expressed as a combination of function invocations in a high-level language. Because necessary code and data are not statically known prior to execution, they must be moved into the cluster at runtime. An obvious way of doing this is to translate function invocations into self-contained executable programs and run them as usual, but this brings a hefty performance penalty: a function invocation now needs to piggyback its context with extra code and data to a remote node, and the remote node needs to take extra time to reconstruct the invocation's context before executing it, both detrimental to lightweight short-running functions. A better solution for workflow systems is to treat functions and invocations as first-class abstractions: subsequent invocations of the same function on a worker node should only pay for the cost of context setup once and reuse the context between different invocations. The remaining problems lie in discovering, distributing, and retaining the reusable context among workers. In this paper, we discuss the rationale and design requirement of these mechanisms to support context reuse, and implement them in TaskVine, a data-intensive distributed framework and execution engine. Our results from executing a large-scale neural network inference application and a molecular design application show that treating functions and invocations as first-class abstractions reduces the execution time of the applications by 94.5% and 26.9%, respectively.
- 3 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 122-134
Open AccessAccelerating Function-Centric Applications by Discovering, Distributing, and Retaining Reusable Context in Workflow Systems
Workflow systems provide a convenient way for users to write large-scale applications by composing independent tasks into large graphs that can be executed concurrently on high-performance clusters. In many newer workflow systems, tasks are often expressed as a combination of function invocations in a high-level language. Because necessary code and data are not statically known prior to execution, they must be moved into the cluster at runtime. An obvious way of doing this is to translate function invocations into self-contained executable programs and run them as usual, but this brings a hefty performance penalty: a function invocation now needs to piggyback its context with extra code and data to a remote node, and the remote node needs to take extra time to reconstruct the invocation's context before executing it, both detrimental to lightweight short-running functions. A better solution for workflow systems is to treat functions and invocations as first-class abstractions: subsequent invocations of the same function on a worker node should only pay for the cost of context setup once and reuse the context between different invocations. The remaining problems lie in discovering, distributing, and retaining the reusable context among workers. In this paper, we discuss the rationale and design requirement of these mechanisms to support context reuse, and implement them in TaskVine, a data-intensive distributed framework and execution engine. Our results from executing a large-scale neural network inference application and a molecular design application show that treating functions and invocations as first-class abstractions reduces the execution time of the applications by 94.5% and 26.9%, respectively.
- Open Access
- 12 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 1978-1988
TaskVine: Managing In-Cluster Storage for High-Throughput Data Intensive Workflows
Many scientific applications are expressed as high-throughput workflows that consist of large graphs of data assets and tasks to be executed on large parallel and distributed systems. A challenge in executing these workflows is managing data: both datasets and software must be efficiently distributed to cluster nodes; intermediate data must be conveyed between tasks; output data must be delivered to its destination. Scaling problems result when these actions are performed in an uncoordinated manner on a shared filesystem. To address this problem, we introduce TaskVine: a system for exploiting the aggregate local storage and network capacity of a large cluster. TaskVine tracks the lifetime of data in a workflow –from archival sources to final outputs– making use of local storage to distribute, and re-use data wherever possible. We describe the architecture and novel capabilities of TaskVine, and demonstrate its use with applications in genomics, high energy physics, molecular dynamics, and machine learning.
- 12 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 1978-1988
Open AccessTaskVine: Managing In-Cluster Storage for High-Throughput Data Intensive Workflows
Many scientific applications are expressed as high-throughput workflows that consist of large graphs of data assets and tasks to be executed on large parallel and distributed systems. A challenge in executing these workflows is managing data: both datasets and software must be efficiently distributed to cluster nodes; intermediate data must be conveyed between tasks; output data must be delivered to its destination. Scaling problems result when these actions are performed in an uncoordinated manner on a shared filesystem. To address this problem, we introduce TaskVine: a system for exploiting the aggregate local storage and network capacity of a large cluster. TaskVine tracks the lifetime of data in a workflow –from archival sources to final outputs– making use of local storage to distribute, and re-use data wherever possible. We describe the architecture and novel capabilities of TaskVine, and demonstrate its use with applications in genomics, high energy physics, molecular dynamics, and machine learning.
- 1 March 1992
- Published by Elsevier in IFAC Proceedings Volumes
- Vol. 25 (2) , 303-306
Measurement of Cellular Differentiation of Filamentous Fungi using Image Analysis
- 1 March 1992
- Published by Elsevier in IFAC Proceedings Volumes
- Vol. 25 (2) , 303-306
Measurement of Cellular Differentiation of Filamentous Fungi using Image Analysis