A Metropolitan Area Infrastructure for Data Intensive Science
Authors:
- David AbramsonDADavid Abramson
- Jake CarrollJCJake Carroll
- Chao JinCJChao Jin
- Michael MallonMMMichael Mallon
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 238-247
Abstract
The increasing amount of data being collected from simulations, instruments and sensors creates challenges for existing e-Science infrastructure. In particular, it requires new ways of storing, distributing and processing data in order to cope with both the volume and velocity of the data. The University of Queensland has recently designed and deployed MeDiCI, a data fabric that spans the metropolitan area and provides seamless access to data regardless of where it is created, manipulated and archived. MeDiCI is novel in that it exploits temporal and spatial locality to move data on demand in an automated manner. This means that data only needs to reside locally in high speed storage whilst being manipulated, and it can be archived transparently in high capacity, but slower, technologies at other times. MeDiCI is built on commercially available technologies. In this paper, we describe these innovations and present some early results.Keywords
This publication has 11 references indexed in Scilit:
- SPANStore2013
- SpannerACM Transactions on Computer Systems, 2013
- The Fourth Paradigm – Data-Intensive Scientific Discovery2012
- The Akamai networkACM SIGOPS Operating Systems Review, 2010
- Chapman & Hall/CRC Computational ScienceCombinatorial Scientific Computing, 2010
- The Hadoop Distributed File System2010
- The XtreemFS architecture—a case for object‐based file systems in GridsConcurrency and Computation: Practice and Experience, 2008
- The Globus Striped GridFTP Framework and Server2005
- The Google file system2003
- OceanStore2000