The Large Scale Data Facility: Data Intensive Computing for Scientific Experiments
Authors:
- Ariel O. GarciaAGAriel O. Garcia
- Serguei BourovSBSerguei Bourov
- Ahmad HammadAHAhmad Hammad
- Jos van WezelJWJos van Wezel
- Bernhard NeumairBNBernhard Neumair
- Achim StreitASAchim Streit
- Volker HartmannVHVolker Hartmann
- Thomas JejkalTJThomas Jejkal
- Patrick NeubergerPNPatrick Neuberger
- Rainer StotzkaRSRainer Stotzka
- 1 May 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1467-1474
Abstract
The Large Scale Data Facility (LSDF) at the Karlsruhe Institute of Technology was started end of 2009 with the aim of supporting the growing requirements of data intensive experiments. In close cooperation with the involved scientific communities, the LSDF provides them not only with adequate storage space but with a directly attached analysis farm and - more importantly - with value added services for their big scientific data-sets. Analysis workflows are supported through the mixed Hadoop and Open Nebula Cloud environments directly attached to the storage, and enable the efficient processing of the experimental data. Metadata handling is a central part of the LSDF, where a metadata repository, community specific metadata schemes, graphical tools, and APIs were developed for accessing and efficiently organizing the stored data-sets.Keywords
This publication has 3 references indexed in Scilit:
- Automated high-throughput mapping of promoter-enhancer interactions in zebrafish embryosNature Methods, 2009
- Scientific workflow design for mere mortalsFuture Generation Computer Systems, 2009
- Three-dimensional display in nuclear medicineIEEE Transactions on Medical Imaging, 1989