Universal Data Junction: A Transport Layer for Data Driven Workflows
Open Access
- 26 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-9
Abstract
A novel transport library for the efficient coupling of applications through their data dependencies is presented. The design is driven by the intent to require minimal changes to existing scientific applications and to declare the data objects that are meaningful for other applications for read and write as well as to perform transparent transport including automatic redistribution of parallel data structures, thus permitting seamless coupling of applications in workflows. The actual transport can be selected at run time, and can exploit a variety of data exchange methods, including MPI, Dataspaces, Ceph Rados, CRAY Datawarp, and a POSIX file system. For the case of MPI transport, the library is used to implement the first stage of a co-working visualization pipeline for CP2K and results show a significant advantage compared to a filesystem based approach.Keywords
This publication has 7 references indexed in Scilit:
- A Middleware Supporting Data Movement in Complex and Software-Defined Storage and Memory Architectures2021
- ADIOS 2: The Adaptable Input Output System. A framework for high-performance data managementSoftwareX, 2020
- CP2K: An electronic structure and molecular dynamics software package - Quickstep: Efficient and accurate electronic structure calculationsThe Journal of Chemical Physics, 2020
- ASPEN: An Efficient Algorithm for Data Redistribution Between Producer and Consumer Grids2018
- DataSpaces2010
- Parallel netCDF2003
- ScaLAPACK Users' Guide1997