Pattern-aware file reorganization in MPI-IO
Authors:
Open Access
- 13 November 2011
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 43-48
Abstract
Scientific computing is becoming more data-intensive; however I/O throughput is not growing at the same rate. MPI-IO and parallel file systems are expected to help bridge the gap by increasing data access parallelism. Compared to traditional I/O systems, some factors are more important in parallel I/O system in order to achieve better performance, such as the number of requests and contiguousness of accesses. The variation of these factors can lead to significant differences in performance. Programmers usually arrange data in a logical fashion for ease of programming and data manipulation; however, this may not be ideal for parallel I/O systems. Without taking into account the organization of file and behavior of the I/O system, the performance may be badly degraded. In this paper, a novel method of reorganizing files in I/O middleware level is proposed, which takes into account the access patterns. By placing data in a way favoring the parallel I/O system, gains of up to two orders of magnitudes in reading and up to one order of magnitude in writing were observed with spinning disks and solid-state disks.Keywords
This publication has 5 references indexed in Scilit:
- IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination2010
- PLFS2009
- Bridging the Gap Between Parallel File Systems and Local File Systems: A Case Study with PVFS2008
- Profile-guided I/O partitioning2003
- Input/output characteristics of scalable parallel applications1995