Hierarchical Caches for Grid Workflows
Authors:
- David ChiuDCDavid Chiu
- Gagan AgrawalGAGagan Agrawal
- 1 January 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 228-235
Abstract
From personal software to advanced systems, caching mechanisms have steadfastly been a ubiquitous means for reducing workloads. It is no surprise, then, that under the grid and cluster paradigms, middlewares and other large-scale applications often seek caching solutions. Among these distributed applications, scientific workflow management systems have gained ground towards mitigating the often painstaking process of composing sequences of scientific data sets and services to derive virtual data. In the past, workflow managers have relied on low-level system cache for reuse support. But in distributed query intensive environments, where high volumes of intermediate virtual data can potentially be stored anywhere on the grid, a novel cache structure is needed to efficiently facilitate workflow planning. In this paper, we describe an approach to combat the challenges of maintaining large, fast virtual data caches for workflow composition. A hierarchical structure is proposed for indexing scientific data with spatiotemporal annotations across grid nodes. Our experimental results show that our hierarchical index is scalable and outperforms a centralized indexing scheme by an exponential factor in query intensive environments.Keywords
This publication has 20 references indexed in Scilit:
- A novel domain oriented approach for scientific Grid workflow composition2008
- A Middleware for Developing and Deploying Scalable Remote Mining Services2008
- Provenance trails in the Wings/Pegasus systemConcurrency and Computation: Practice and Experience, 2007
- Triana: a graphical Web service composition and execution toolkit2004
- Accurate unlexicalized parsing2003
- Condor-G: a computation management agent for multi-institutional grids2002
- Chimera: a virtual data system for representing, querying, and automating data derivation2002
- LRFU: a spectrum of policies that subsumes the least recently used and least frequently used policiesIEEE Transactions on Computers, 2001
- Analysis of the clustering properties of the Hilbert space-filling curveIEEE Transactions on Knowledge and Data Engineering, 2001
- R-trees1984