ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum
Authors:
- 31 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 221-233
Abstract
Modern scientific workflows require hybrid infrastructures combining numerous decentralized resources on the IoT/Edge interconnected to Cloud/HPC systems (aka the Computing Continuum) to enable their optimized execution. Understanding and optimizing the performance of such complex Edge-to-Cloud workflows is challenging. Capturing the provenance of key performance indicators, with their related data and processes, may assist in understanding and optimizing workflow executions. However, the capture overhead can be prohibitive, particularly in resource-constrained devices, such as the ones on the IoT/Edge.To address this challenge, based on a performance analysis of existing systems, we propose ProvLight, a tool to enable efficient provenance capture on the IoT/Edge. We leverage simplified data models, data compression and grouping, and lightweight transmission protocols to reduce overheads. We further integrate ProvLight into the E2Clab framework to enable workflow provenance capture across the Edge-to-Cloud Continuum. This integration makes E2Clab a promising platform for the performance optimization of applications through reproducible experiments.We validate ProvLight at a large scale with synthetic workloads on 64 real-life IoT/Edge devices in the FIT IoT LAB testbed. Evaluations show that ProvLight outperforms state-of-the-art systems like ProvLake and DfAnalyzer in resource-constrained devices. ProvLight is 26—37x faster to capture and transmit provenance data; uses 5—7x less CPU; 2x less memory; transmits 2x less data; and consumes 2—2.5x less energy. ProvLight [1] and E2Clab [2] are available as open-source tools.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- A survey on provenance: What for? What form? What from?The VLDB Journal, 2017
- Towards a provenance collection framework for Internet of Things devices2017
- An Approach to Standalone Provenance Systems for Big Social Provenance Data2016
- FIT IoT-LAB: A large scale open experimental IoT testbed2015
- Pegasus, a workflow management system for science automationFuture Generation Computer Systems, 2015
- The W3C PROV family of specifications for modelling provenance metadata2013
- Encyclopedia of Database Systems2009
- Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance Tracking: The CyberShake Example2006
- Scientific workflow management and the Kepler systemConcurrency and Computation: Practice and Experience, 2005
- Taverna: lessons in creating a workflow environment for the life sciencesConcurrency and Computation: Practice and Experience, 2005