Exploiting Analytics Shipping with Virtualized MapReduce on HPC Backend Storage Servers
Authors:
- 7 January 2015
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 27 (1) , 185-196
Abstract
Large-scale scientific applications on High-Performance Computing (HPC) systems are generating a colossal amount of data that need to be analyzed in a timely manner for new knowledge, but are too costly to transfer due to their sheer size. Many HPC systems have catered to in situ analytics solutions that can analyze temporary datasets as they are generated, i.e., without storing to long-term storage media. However, there is still an open question on how to conduct efficient analytics of permanent datasets that have been stored to the backend persistent storage because of their long-term value. To fill the void, we exploit the analytics shipping model for fast analysis of large-scale scientific datasets on HPC backend storage servers. Through an efficient integration of MapReduce and the popular Lustre storage system, we have developed a Virtualized Analytics Shipping (VAS) framework that can ship MapReduce programs to Lustre storage servers. The VAS framework includes three component techniques: (a) virtualized analytics shipping with fast network and disk I/O; (b) stripe-aligned data distribution and task scheduling and (c) pipelined intermediate data merging and reducing. The first technique provides necessary isolation between MapReduce analytics and Lustre I/O services. The second and third techniques optimize MapReduce on Lustre and avoid explicit shuffling. Our performance evaluation demonstrates that VAS offers an exemplary implementation of analytics shipping and delivers fast and virtualized MapReduce programs on backend Lustre storage servers.Keywords
Funding Information
- Intel and Lawrence Livermore National Laboratory
- NSF (1059376, 1320016, 1340947, 1432892)
This publication has 11 references indexed in Scilit:
- IndexFS: Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion2014
- Will They Blend?: Exploring Big Data Computation Atop Traditional HPC NAS Storage2014
- In-situ Feature-Based Objects Tracking for Large-Scale Scientific Simulations2012
- Combining in-situ and in-transit processing to enable extreme-scale scientific analysis2012
- On the duality of data-intensive file system design2011
- The Hadoop Distributed File System2010
- MapReduceCommunications of the ACM, 2008
- Evaluation of active storage strategies for the lustre parallel file system2007
- Exploiting Lustre File Joining for Effective Collective IO2007
- The Nas Parallel BenchmarksThe International Journal of Supercomputing Applications, 1991