Stork data scheduler: mitigating the data bottleneck in e-Science
Authors:
- 28 August 2011
- journal article
- Published by The Royal Society in Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Vol. 369 (1949) , 3254-3267
Abstract
In this paper, we present the Stork data scheduler as a solution for mitigating the data bottleneck in e-Science and data-intensive scientific discovery. Stork focuses on planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked inputs/outputs for petascale distributed e-Science applications. Unlike existing approaches, Stork treats data resources and the tasks related to data access and movement as first-class entities just like computational resources and compute tasks, and not simply the side-effect of computation. Stork provides unique features such as aggregation of data transfer jobs considering their source and destination addresses, and an application-level throughput estimation and optimization service. We describe how these two features are implemented in Stork and their effects on end-to-end data transfer performance.Keywords
This publication has 6 references indexed in Scilit:
- Performance engineering in data GridsConcurrency and Computation: Practice and Experience, 2005
- Wide area data replication for scientific collaborations2005
- Computation Scheduling and Data Replication Algorithms for Data Grids2004
- Data management and transfer in high-performance computational grid environmentsParallel Computing, 2002
- Computing and Data Grids for Science and Engineering2000
- Differentiated end-to-end Internet services using a weighted proportional fair sharing TCPACM SIGCOMM Computer Communication Review, 1998