Building the Trident Scientific Workflow Workbench for Data Management in the Cloud
Authors:
- 1 October 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 41-50
Abstract
Scientific workflows have gained popularity for modeling and executing in silico experiments by scientists for problem-solving. These workflows primarily engage in computation and data transformation tasks to perform scientific analysis in the Science Cloud. Increasingly workflows are gaining use in managing the scientific data when they arrive from external sensors and are prepared for becoming science ready and available for use in the Cloud. While not directly part of the scientific analysis, these workflows operating behind the Cloud on behalf of the -data valets¿ play an important role in end-to-end management of scientific data products. They share several features with traditional scientific workflows: both are data intensive and use Cloud resources. However, they also differ in significant respects, for example, in the reliability required, scheduling constraints and the use of provenance collected. In this article, we investigate these two classes of workflows - Science Application workflows and Data Preparation workflows - and use these to drive common and distinct requirements from workflow systems for eScience in the Cloud. We use workflow examples from two collaborations, the NEPTUNE oceanography project and the Pan-STARRS astronomy project, to draw out our comparison. Our analysis of these workflows classes can guide the evolution of workflow systems to support emerging applications in the Cloud and the Trident Scientific Workbench is one such workflow system that has directly benefitted from this to meet the needs of these two eScience projects.Keywords
This publication has 11 references indexed in Scilit:
- The Trident Scientific Workflow Workbench2008
- Examining Statistics of Workflow Evolution Provenance: A First StudyLecture Notes in Computer Science, 2008
- Semantic Provenance for eScience: Managing the Deluge of Scientific DataIEEE Internet Computing, 2008
- Exploring ocean dataACM SIGMOD Record, 2008
- All-pairs: An abstraction for data-intensive cloud computing2008 IEEE International Symposium on Parallel and Distributed Processing, 2008
- Issues in Automatic Provenance CollectionLecture Notes in Computer Science, 2006
- Towards a Quality Model for Effective Data Selection in Collaboratories2006
- A survey of data provenance in e-scienceACM SIGMOD Record, 2005
- Active Management of Scientific DataIEEE Internet Computing, 2005
- Project Pan-STARRS and the Outer Solar SystemEarth, Moon, and Planets, 2003