Model-Driven Data Layout Selection for Improving Read Performance
Authors:
- Jialin LiuJLJialin Liu
- Surendra BynaSBSurendra Byna
- Bin DongBDBin Dong
- Kesheng WuKesheng Wu
- Yong ChenYCYong Chen
- 1 May 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1708-1716
Abstract
Performance of reading scientific data from a parallel file system depends on the organization of data on physical storage devices. Data is often immutable after producers of data, such as large-scale simulations, experiments, and observations, write the data to the parallel file system. As a result, read performance of data analysis tasks is often slow when the read pattern does not conform with the original organization of the data. For example, reading small noncontiguous chunks of data from a large array is many times slower than reading the same size of contiguous chunks of data. Towards improving the data read performance during analysis phase, we are developing the Scientific Data Services (SDS) framework for automatically reorganizing previously written data to conform with the known read patterns. In this paper, we introduce a model-driven strategy for selecting the data layouts that benefit the performance of different read patterns. We have developed a parallel I/O model based on the striping parameters on Lustre file system and the block-level striping on RAID-based disks within an Object Storage Target (OST) of Lustre. We have applied the model to reorganize large 3D array datasets on a Cray XE6 platform and achieved 9X to 128X improvement in accessing the reorganized data compared to reading the data in its original layout.Keywords
This publication has 15 references indexed in Scilit:
- Pattern-Direct and Layout-Aware Replication Scheme for Parallel I/O Systems2013
- SMART-IO: SysteM-AwaRe Two-Level Data Organization for Efficient Scientific Analytics2012
- Cost-intelligent application-specific data layout optimization for parallel file systemsCluster Computing, 2012
- EDO: Improving Read Performance for Scientific Applications through Elastic Data Organization2011
- Six degrees of scientific data2011
- PLFS2009
- Characterizing and predicting the I/O performance of HPC applications using a parameterized synthetic benchmark2008
- Understanding Interaction Models: Improving Empirical AnalysesPolitical Analysis, 2006
- Efficient clustering of large EST data sets on parallel computersNucleic Acids Research, 2003
- Characterizing parallel file-access patterns on a large-scale multiprocessor2002