A lightweight I/O scheme to facilitate spatial and temporal queries of scientific data analytics
Authors:
- 1 May 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
In the era of petascale computing, more scientific applications are being deployed on leadership scale computing platforms to enhance the scientific productivity. Many I/O techniques have been designed to address the growing I/O bottleneck on large-scale systems by handling massive scientific data in a holistic manner. While such techniques have been leveraged in a wide range of applications, they have not been shown as adequate for many mission critical applications, particularly in data postprocessing stage. One of the examples is that some scientific applications generate datasets composed of a vast amount of small data elements that are organized along many spatial and temporal dimensions but require sophisticated data analytics on one or more dimensions. Including such dimensional knowledge into data organization can be beneficial to the efficiency of data post-processing, which is often missing from exiting I/O techniques. In this study, we propose a novel I/O scheme named STAR (Spatial and Temporal AggRegation) to enable high performance data queries for scientific analytics. STAR is able to dive into the massive data, identify the spatial and temporal relationships among data variables, and accordingly organize them into an optimized multi-dimensional data structure before storing to the storage. This technique not only facilitates the common access patterns of data analytics, but also further reduces the application turnaround time. In particular, STAR is able to enable efficient data queries along the time dimension, a practice common in scientific analytics but not yet supported by existing I/O techniques. In our case study with a critical climate modeling application GEOS-5, the experimental results on Jaguar supercomputer demonstrate an improvement up to 73 times for the read performance compared to the original I/O method.Keywords
This publication has 24 references indexed in Scilit:
- A case study for scientific I/O: improving the FLASH astrophysics codeComputational Science & Discovery, 2012
- Six degrees of scientific data2011
- Managing Variability in the IO Performance of Petascale Storage Systems2010
- PreDatA – preparatory data analytics on peta-scale machines2010
- Adaptable, metadata rich IO methods for portable high performance IO2009
- Optimal chunking of large multidimensional arrays for data warehousing2007
- Parallel netCDF2003
- Improving collective I/O performance using threads2003
- Disk-directed I/O for MIMD multiprocessorsACM Transactions on Computer Systems, 1997
- Ueber die stetige Abbildung einer Line auf ein Fl chenst ckMathematische Annalen, 1891