Fault tolerant MapReduce-MPI for HPC clusters
Authors:
Open Access
- 15 November 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Abstract
Building MapReduce applications using the Message-Passing Interface (MPI) enables us to exploit the performance of large HPC clusters for big data analytics. However, due to the lacking of native fault tolerance support in MPI and the incompatibility between the MapReduce fault tolerance model and HPC schedulers, it is very hard to provide a fault tolerant MapReduce runtime for HPC clusters. We propose and develop FT-MRMPI, the first fault tolerant MapReduce framework on MPI for HPC clusters. We discover a unique way to perform failure detection and recovery by exploiting the current MPI semantics and the new proposal of user-level failure mitigation. We design and develop the checkpoint/restart model for fault tolerant MapReduce in MPI. We further tailor the detect/resume model to conserve work for more efficient fault tolerance. The experimental results on a 256-node HPC cluster show that FT-MRMPI effectively masks failures and reduces the job completion time by 39%.Keywords
Funding Information
- National Science Foundation (CNS-0844983, CNS-1217979, CNS-1422119)
- U.S. Department of Energy (DE- AC02-06CH11357)
This publication has 27 references indexed in Scilit:
- Hadoop's adolescenceProceedings of the VLDB Endowment, 2013
- MRBS: Towards Dependability Benchmarking for Hadoop MapReduce2013
- A Checkpoint-on-Failure Protocol for Algorithm-Based Recovery in Standard MPILecture Notes in Computer Science, 2012
- Resource Provisioning Framework for MapReduce Jobs with Performance Goals2011
- Fault Tolerance in Petascale/ Exascale Systems: Current Knowledge, Challenges and Research OpportunitiesThe International Journal of High Performance Computing Applications, 2009
- Understanding failures in petascale computersJournal of Physics: Conference Series, 2007
- Fault Tolerance in Message Passing Interface ProgramsThe International Journal of High Performance Computing Applications, 2004
- MPICH-V: Toward a Scalable Fault Tolerant MPI for Volatile Nodes2002
- FT-MPI: Fault Tolerant MPI, Supporting Dynamic Applications in a Dynamic World2000
- Gang scheduling performance benefits for fine-grain synchronizationJournal of Parallel and Distributed Computing, 1992