Best Practices and Lessons Learned from Deploying and Operating Large-Scale Data-Centric Parallel File Systems
Authors:
- Sarp OralSOSarp Oral
- James SimmonsJSJames Simmons
- Jason HillJHJason Hill
- Dustin LevermanDLDustin Leverman
- Feiyi WangFWFeiyi Wang
- Matt EzellMEMatt Ezell
- Ross MillerRMRoss Miller
- Douglas FullerDFDouglas Fuller
- Raghul GunasekaranRGRaghul Gunasekaran
- Youngjae KimYKYoungjae Kim
- Saurabh GuptaSGSaurabh Gupta
- Devesh Tiwari Sudharshan S. VazhkudaiDVDevesh Tiwari Sudharshan S. Vazhkudai
- 1 November 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 21674329,p. 217-228
Abstract
The Oak Ridge Leadership Computing Facility (OLCF) has deployed multiple large-scale parallel file systems (PFS) to support its operations. During this process, OLCF acquired significant expertise in large-scale storage system design, file system software development, technology evaluation, benchmarking, procurement, deployment, and operational practices. Based on the lessons learned from each new PFS deployment, OLCF improved its operating procedures, and strategies. This paper provides an account of our experience and lessons learned in acquiring, deploying, and operating large-scale parallel file systems. We believe that these lessons will be useful to the wider HPC community.Keywords
This publication has 5 references indexed in Scilit:
- Improving large-scale storage system performance via topology-aware and balanced data placement2014
- Efficient and Scalable Retrieval Techniques for Global File Properties2013
- Enhancing I/O throughput via efficient routing and placement for large-scale parallel file systems2011
- Workload characterization of a leadership class storage cluster2010
- Direct numerical simulation of turbulent combustion: fundamental insights towards predictive modelsJournal of Physics: Conference Series, 2005