Barry Sly-Delgado
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Barry Sly-Delgado has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Andrew Hennessee
University of Notre Dame, South Bend, Indiana, USA
1 shared publication - Benjamin Tovar
Department of Computer Science and Engineering, University of Notre Dame, 6111 Notre Dame, Indiana United States
1 shared publication - Colin Thomas
Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, Indiana, USA
1 shared publication - David Simonetti
University of Notre Dame, United States of America
1 shared publication - Douglas Thain
University of Notre Dame
1 shared publication - Thanh Son Phung
University of Notre Dame
1 shared publication
Latest Publications
- Open Access
- 12 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 1978-1988
TaskVine: Managing In-Cluster Storage for High-Throughput Data Intensive Workflows
Many scientific applications are expressed as high-throughput workflows that consist of large graphs of data assets and tasks to be executed on large parallel and distributed systems. A challenge in executing these workflows is managing data: both datasets and software must be efficiently distributed to cluster nodes; intermediate data must be conveyed between tasks; output data must be delivered to its destination. Scaling problems result when these actions are performed in an uncoordinated manner on a shared filesystem. To address this problem, we introduce TaskVine: a system for exploiting the aggregate local storage and network capacity of a large cluster. TaskVine tracks the lifetime of data in a workflow –from archival sources to final outputs– making use of local storage to distribute, and re-use data wherever possible. We describe the architecture and novel capabilities of TaskVine, and demonstrate its use with applications in genomics, high energy physics, molecular dynamics, and machine learning.
- 12 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 1978-1988
Open AccessTaskVine: Managing In-Cluster Storage for High-Throughput Data Intensive Workflows
Many scientific applications are expressed as high-throughput workflows that consist of large graphs of data assets and tasks to be executed on large parallel and distributed systems. A challenge in executing these workflows is managing data: both datasets and software must be efficiently distributed to cluster nodes; intermediate data must be conveyed between tasks; output data must be delivered to its destination. Scaling problems result when these actions are performed in an uncoordinated manner on a shared filesystem. To address this problem, we introduce TaskVine: a system for exploiting the aggregate local storage and network capacity of a large cluster. TaskVine tracks the lifetime of data in a workflow –from archival sources to final outputs– making use of local storage to distribute, and re-use data wherever possible. We describe the architecture and novel capabilities of TaskVine, and demonstrate its use with applications in genomics, high energy physics, molecular dynamics, and machine learning.