FTHP-MPI: Towards Providing Replication-based Fault Tolerance in a Fault-Intolerant Native MPI Library
Preprint
Abstract
Faults in high-performance systems are expected to be very large in the current exascale computing era. To compensate for a higher failure rate, the standard checkpoint/restart technique would need to create checkpoints at a much higher frequency resulting in an excessive amount of overhead which would not be sustainable for many scientific applications. To improve application efficiency in such high failure environments, the mechanism of replication of MPI processes was proposed. Replication allows for fast recovery from failures by simply dropping the failed processes and using their replicas to continue the regular operation of the application. In this paper, we have implemented FTHP-MPI (Fault Tolerance and High Performance MPI), a novel fault-tolerant MPI library that augments checkpoint/restart with replication to provide resilience from failures. The novelty of our work is that it is designed to provide fault tolerance in a native MPI library that does not provide support for fault tolerance. This lets application developers achieve fault tolerance at high failure rates while also using efficient communication protocols in the native MPI libraries that are generally fine-tuned for specific HPC platforms. We have also implemented efficient parallel communication techniques that involve replicas. Our framework deals with the unique challenges of integrating support for checkpointing and partial replication. We conducted experiments emulating the failure rates of exascale computing systems with three applications, HPCG, PIC and CloverLeaf. We show that for large scale systems where the failure intervals are expected to be within a hour, our replication-based library provides higher efficiency and performance than checkpointing-based approaches. We show that under failure-free conditions, the additional overheads due to replication are negligible in our library.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- CRIU - Checkpoint Restore in Userspace for computational simulations and scientific applicationsEPJ Web of Conferences, 2024
- Reinit$^{++}$: Evaluating the Performance of Global-Restart Recovery Methods for MPI Fault Tolerance2020
- Algorithm-Based Fault Tolerance for Dense Matrix Factorizations, Multiple Failures and AccuracyACM Transactions on Parallel Computing, 2015
- Fault Tolerance on Large Scale Systems using Adaptive Process ReplicationIEEE Transactions on Computers, 2014
- A survey of fault tolerance mechanisms and checkpoint/restart implementations for high performance computing systemsThe Journal of Supercomputing, 2013
- Algorithm-based fault tolerance applied to high performance computingJournal of Parallel and Distributed Computing, 2009
- A Large-Scale Study of Failures in High-Performance Computing SystemsIEEE Transactions on Dependable and Secure Computing, 2009
- Replication-Based Fault Tolerance for MPI ApplicationsIEEE Transactions on Parallel and Distributed Systems, 2008
- Berkeley lab checkpoint/restart (BLCR) for Linux clustersJournal of Physics: Conference Series, 2006
- Skeleton PIC codes for parallel computersComputer Physics Communications, 1995