SMART-IO: SysteM-AwaRe Two-Level Data Organization for Efficient Scientific Analytics
Authors:
- Yuan TianYTYuan Tian
- Scott KlaskySKScott Klasky
- Weikuan YuWYWeikuan Yu
- Hasan AbbasiHAHasan Abbasi
- Bin WangBWBin Wang
- Norbert PodhorszkiNPNorbert Podhorszki
- Ray GroutRGRay Grout
- Matthew WolfMWMatthew Wolf
- 1 August 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15267539,p. 181-188
Abstract
Current I/O techniques have pushed the write performance close to the system peak, but they usually overlook the read side of problem. With the mounting needs of scientific discovery, it is important to provide good read performance for many common access patterns. Such demand requires an organization scheme that can effectively utilize the underlying storage system. However, the mismatch between conventional data layout on disk and common scientific access patterns leads to significant performance degradation when a subset of data is accessed. To this end, we design a system-aware Optimized Chunking model, which aims to find an optimized organization that can strike for a good balance between data transfer efficiency and processing overhead. To enable such model for scientific applications, we propose SMART-IO, a two-level data organization framework that can organize the blocks of multidimensional data efficiently. This scheme can adapt data layouts based on data characteristics and underlying storage systems, and enable efficient scientific analytics. Our experimental results demonstrate that SMART-IO can significantly improve the read performance for challenging access patterns, and speed up data analytics. For a mission critical combustion simulation code S3D, Smart-IO achieves up to 72 times speedup for planar reads of a 3-D variable compared to the logically contiguous data layout.Keywords
This publication has 23 references indexed in Scilit:
- Six degrees of scientific data2011
- Managing Variability in the IO Performance of Petascale Storage Systems2010
- PreDatA – preparatory data analytics on peta-scale machines2010
- ...and eat it too2009
- Adaptable, metadata rich IO methods for portable high performance IO2009
- ParColl: Partitioned Collective I/O on the Cray XT2008
- Optimal chunking of large multidimensional arrays for data warehousing2007
- Performance engineering in data GridsConcurrency and Computation: Practice and Experience, 2005
- Clustering high dimensional massive scientific datasets2002
- The Recovery Manager of the System R Database ManagerACM Computing Surveys, 1981