Resiliency of HPC Interconnects: A Case Study of Interconnect Failures and Recovery in Blue Waters
- 9 August 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Dependable and Secure Computing
- Vol. 15 (6) , 915-930
Abstract
Availability of the interconnection network in high-performance computing (HPC) systems is fundamental to sustaining the continuous execution of applications at scale. When failures occur, interconnect recovery mechanisms orchestrate complex operations to recover network connectivity between the nodes. As the scale and design complexity of HPC systems increase, so does the system's susceptibility to failures during execution of interconnect-recovery procedures. This study characterizes the recovery procedures of the Gemini interconnect network, the largest Gemini network built by Cray, on Blue Waters, a 13.3 petaflop supercomputer at the National Center for Supercomputing Applications (NCSA). We propose a propagation model that captures interconnect failures and recovery procedures to help understand types of failures and their propagation in both the system and applications during recovery. The measurements show that recovery procedures occur very frequently and that the unsuccessful execution of recovery procedures, when additional failures occur during recovery, causes system-wide outages (SWOs, 28 out of 101) and application failures (3.4 percent of all running applications).Keywords
Funding Information
- U.S. Department of Energy, Office of Science
- Office of Advanced Scientific Computing Research (2015-02674)
- US National Science Foundation (CNS 13-14891)
- Air Force Research Lab (FA8750-11-2-0084)
- US National Science Foundation (OCI-0725070, ACI-1238993)
This publication has 26 references indexed in Scilit:
- Toward Exascale ResilienceThe International Journal of High Performance Computing Applications, 2009
- Major Computer Science Challenges At ExascaleThe International Journal of High Performance Computing Applications, 2009
- Fault Tolerance in Petascale/ Exascale Systems: Current Knowledge, Challenges and Research OpportunitiesThe International Journal of High Performance Computing Applications, 2009
- Wormhole routing techniques for directly connected multicomputer systemsACM Computing Surveys, 1998
- CHARM++ACM SIGPLAN Notices, 1993
- An adaptive and fault tolerant wormhole routing strategy for k-ary n-cubesIEEE Transactions on Computers, 1991
- Error log analysis: statistical modeling and heuristic trend analysisIEEE Transactions on Reliability, 1990
- Analysis of the increase and decrease algorithms for congestion avoidance in computer networksComputer Networks and ISDN Systems, 1989
- Congestion avoidance and controlACM SIGCOMM Computer Communication Review, 1988
- Virtual cut-through: A new computer communication switching techniqueComputer Networks (1976), 1979