Open Access
- 1 January 2007
- journal article
- research article
- Published by Wiley in Scientific Programming
- Vol. 15 (4) , 249-268
Abstract
In this paper we examine the issue of optimizing disk usage and scheduling large‐scale scientific workflows onto distributed resources where the workflows are data‐intensive, requiring large amounts of data storage, and the resources have limited storage resources. Our approach is two‐fold: we minimize the amount of space a workflow requires during execution by removing data files at runtime when they are no longer needed and we demonstrate that workflows may have to be restructured to reduce the overall data footprint of the workflow. We show the results of our data management and workflow restructuring solutions using a Laser Interferometer Gravitational‐Wave Observatory (LIGO) application and an astronomy application, Montage, running on a large‐scale production grid‐the Open Science Grid. We show that although reducing the data footprint of Montage by 48% can be achieved with dynamic data cleanup techniques, LIGO Scientific Collaboration workflows require additional restructuring to achieve a 56% reduction in data space usage. We also examine the cost of the workflow restructuring in terms of the application′s runtime.Keywords
This publication has 14 references indexed in Scilit:
- Search for gravitational waves from binary inspirals in S3 and S4 LIGO dataPhysical Review D, 2008
- Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance Tracking: The CyberShake Example2006
- Search for gravitational waves from binary black hole inspirals in LIGO dataPhysical Review D, 2006
- A Grid service broker for scheduling e‐Science applications on global data GridsConcurrency and Computation: Practice and Experience, 2005
- Search for gravitational waves from primordial black hole binary coalescences in the galactic haloPhysical Review D, 2005
- Scheduling of scientific workflows in the ASKALON grid environmentACM SIGMOD Record, 2005
- Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed SystemsScientific Programming, 2005
- Workflow Management in Griphyn2004
- Static scheduling algorithms for allocating directed task graphs to multiprocessorsACM Computing Surveys, 1999
- Globus: a Metacomputing Infrastructure ToolkitThe International Journal of Supercomputer Applications and High Performance Computing, 1997