Diaspora: Resilience-Enabling Services for Real-Time Distributed Workflows
Authors:
- 16 September 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-9
Abstract
The need for real-time processing to enable automated decision making and experimental steering has driven a shift from high-performance computing workflows on a centralized system to a distributed approach that integrates remote data sources, edge devices, and diverse compute facilities. Under this paradigm, data can be processed close to the source where it is generated, thus reducing latency and bandwidth usage. System resilience is thus a key challenge, requiring distributed workflows to survive component failures and to meet stringent quality-of-service requirements, which results in the need to mitigate anomalies such as congestion and low availability of resources. To address these challenges, we propose Diaspora, a unified resilience framework that is inspired by event-driven communication patterns used in public clouds. Specifically, we propose an event fabric that extends across sites, facilities, and computations to provide timely, reliable, and accurate information about data, application, and resource status. On top of the event fabric, we build resilience-enabling services that combine QoS-aware data streaming, resilient data views, resilient compute and data resources, and anomaly detection and prediction, all of which collectively enhance workflow resilience for these scientific cases.Keywords
Funding Information
- U.S. Department of Energy
This publication has 80 references indexed in Scilit:
- High-Performance Ptychographic Reconstruction with Federated Facilities2022
- Virtual Infrastructure Twins: Software Testing Platforms for Computing-Instrument Ecosystems2022
- A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management2022
- Enjoy your observability: an industrial survey of microservice tracing and analysisEmpirical Software Engineering, 2021
- Synchrotron Big Data ScienceSmall, 2018
- The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems2008
- Towards a High Performance Implementation of MPI-IO on the Lustre File SystemLecture Notes in Computer Science, 2008
- Virtual data Grid middleware services for data‐intensive scienceConcurrency and Computation: Practice and Experience, 2005
- Distributed computing in practice: the Condor experienceConcurrency and Computation: Practice and Experience, 2005
- Using Run-Time Predictions to Estimate Queue Wait Times and Improve Scheduler Performance1999