An Efficient, Nonintrusive, Log-Based I/O Mechanism for Scientific Simulations on Clusters
- 1 September 2005
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in 2005 IEEE International Conference on Cluster Computing
- No. 15525244,p. 1-10
Abstract
Scientific simulations are often very I/O intensive, requiring high I/O bandwidth to store the data generated by the simulation. Traditional supercomputers have specialized I/O systems with multiple I/O nodes and specialized interconnects to handle such high I/O loads. However, with the increased availability of inexpensive clusters of workstations, more and more simulations are now run on clusters. Unfortunately, cluster supercomputers are usually not very well equipped for I/O, making I/O a serious bottleneck for such applications. To address this problem, we propose log-based I/O (LBIO), an approach that can substantially increase the I/O performance of simulations on clusters by utilizing free space on the cluster's local disks to stage data on its way to remote storage. LBIO uses local disks to create a log of all I/O calls, and uses a background thread to replay the log at the rate that best utilizes the server and network resources. LBIO is implemented as an easy-to-use, non-intrusive library - a user can turn on LBIO by adding a single initialization call to the simulation code. LBIO also works with existing scientific I/O libraries like HDF, as well as collective libraries like ROMIO. Our performance studies on microbenchmarks and a real-world scientific simulation code show that LBIO can provide upto 35% improvement in I/O performance for raw I/O and over 50% for I/O through libraries like ROMIO or HDFKeywords
This publication has 6 references indexed in Scilit:
- Improving collective I/O performance using threads2003
- Whole System Simulation of Solid Propellant Rockets2002
- Characteristics of scalability and their impact on performance2000
- On implementing MPI-IO portably and with high performance1999
- Data sieving and collective I/O in ROMIO1999
- The behavior of Ethernet-like computer communications networks1979