RISE: Reducing I/O Contention in Staging-based Extreme-Scale In-situ Workflows
Authors:
- 1 September 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 146-156
Abstract
While in-situ workflow formulations have addressed some of the data-related challenges associated with extreme-scale scientific workflows, these workflows involve complex interactions and different modes of data exchange. In the context of increasing system complexity, such workflows present significant resource management challenges, requiring complex cost-performance tradeoffs. This paper presents RISE, an intelligent staging-based data management middleware, which builds on the DataSpaces framework and performs intelligent scheduling of data management operations to reduce I/O contention. In RISE, data are always written immediately to local buffers to reduce the effect of the transfer impact upon application performance. RISE identifies applications’ data access patterns and moves data towards data consumers only when the network is expected to be idle, reducing the impact of asynchronous background data movement upon critical data read/write requests. We experimentally demonstrate that RISE can take advantage of staging nodes to offload data during writes without degrading application data movement performance.Keywords
Funding Information
- Office of Science
This publication has 31 references indexed in Scilit:
- Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows2015
- Opportunities for Nonvolatile Memory Systems in Extreme-Scale High-Performance ComputingComputing in Science & Engineering, 2015
- ActiveSpaces: Exploring dynamic code deployment for extreme scale data processingConcurrency and Computation: Practice and Experience, 2014
- DataSpaces: an interaction and coordination framework for coupled simulation workflowsCluster Computing, 2011
- DataStager: scalable data staging services for petascale applicationsCluster Computing, 2010
- Full-f gyrokinetic particle simulation of centrally heated global ITG turbulence from magnetic axis to edge pedestal top in a realistic tokamak geometryNuclear Fusion, 2009
- Terascale direct numerical simulations of turbulent combustion using S3DComputational Science & Discovery, 2009
- Scientific workflow management and the Kepler systemConcurrency and Computation: Practice and Experience, 2005
- Taverna: a tool for the composition and enactment of bioinformatics workflowsBioinformatics, 2004
- Pegasus: Mapping Scientific Workflows onto the Grid2004