DAOS and Friends: A Proposal for an Exascale Storage System
Authors:
- Jay LofsteadJLJay Lofstead
- Ivo JimenezIJIvo Jimenez
- Carlos MaltzahnCMCarlos Maltzahn
- Quincey KoziolQKQuincey Koziol
- John BentJBJohn Bent
- Eric BartonEBEric Barton
- 16 March 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 585-596
Abstract
The DOE Extreme-Scale Technology Acceleration Fast Forward Storage and IO Stack project is going to have significant impact on storage systems design within and beyond the HPC community. With phase two of the project starting, it is an excellent opportunity to explore the complete design and how it will address the needs of extreme scale platforms. This paper examines each layer of the proposed stack in some detail along with cross-cutting topics, such as transactions and metadata management. This paper not only provides a timely summary of important aspects of the design specifications but also captures the underlying reasoning that is not available elsewhere. We encourage the broader community to understand the design, intent, and future directions to foster discussion guiding phase two and the ultimate production storage stack based on this work. An initial performance evaluation of the early prototype implementation is also provided to validate the presented design.Keywords
This publication has 26 references indexed in Scilit:
- Efficient, Failure Resilient Transactions for Parallel and Distributed Computing2014
- Efficient transactions for parallel data movement2013
- Performance and scalability evaluation of the Ceph parallel file system2013
- Mercury: Enabling remote procedure call for high-performance computing2013
- BTRFSACM Transactions on Storage, 2013
- GraphBuilder2013
- D2T: Doubly Distributed Transactions for High Performance and Distributed Computing2012
- An overview of the HDF5 technology suite and its applications2011
- The Hadoop Distributed File System2010
- The Google file system2003