Design and Evaluation of Network Topology-/Speed- Aware Broadcast Algorithms for InfiniBand Clusters
Authors:
- H. SubramoniHSH. Subramoni
- K. KandallaKKK. Kandalla
- J. VienneJVJ. Vienne
- S. SurSSS. Sur
- B. BarthBBB. Barth
- K. TomkoKTK. Tomko
- R. MclayRMR. Mclay
- K. SchulzKSK. Schulz
- D.K. PandaDPD.K. Panda
- 1 September 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15525244,p. 317-325
Abstract
It is an established fact that the network topology can have an impact on the performance of scientific parallel applications. However, little work has been done to design an easy to use solution inside a communication library supporting a parallel programming model where the complexities of making the application performance network topology agnostic is hidden from the end user. Similarly, the rapid improvements in networking technology and speed are resulting in many commodity clusters becoming heterogeneous, with respect to networking speed. For example, switches and adapters belonging to different generations (SDR - 8 Gbps, DDR - 16 Gbps and QDR - 36 Gbps speeds in InfiniBand) are integrated into a single system. This leads to an additional challenge to make the communication library aware of the performance implications of heterogeneous link speeds. Accordingly, the communication library can perform optimizations taking link speed into account. In this paper, we propose a framework to automatically detect the topology and speed of an InfiniBand network and make it available to users through an easy to use interface. We also make design changes inside the MPI library to dynamically query this topology detection service and to form a topology model of the underlying network. We have redesigned the broadcast algorithm to take into account this network topology information and dynamically adapt the communication pattern to best fit the characteristics of the underlying network. To the best of our knowledge, this is the first such work for InfiniBand clusters. Our experimental results show that, for large homogeneous systems and large message sizes, we get up to 14% improvement in the latency of the broadcast operation using our proposed network topology-aware scheme over the default scheme at the micro-benchmark level. At the application level, the proposed framework delivers up to 8% improvement in total application run-time especially as job size scales up. The proposed network speed-aware algorithms are able to attain micro-benchmark performance on the heterogeneous SDR-DDR InfiniBand cluster to perform on par with runs on the DDR only portion of the cluster for small to medium sized messages. We also demonstrate that the network speed aware algorithms perform 70% to 100% better than the naive algorithms when both are run on the heterogeneous SDR-DDR InfiniBand cluster.Keywords
This publication has 17 references indexed in Scilit:
- Optimizing communication for Charm++ applications by reducing network contentionConcurrency and Computation: Practice and Experience, 2010
- Understanding Application Performance via Micro-benchmarks on Three Large Supercomputers: Intrepid, Ranger and JaguarThe International Journal of High Performance Computing Applications, 2010
- An evaluative study on the effect of contention on message latencies in large supercomputers2009
- Designing multi-leader-based Allgather algorithms for multi-core clusters2009
- Large-Scale Neighbor-Joining with NINJA2009
- MPI Support for Multi-core Architectures: Optimized Shared Memory Collectives2008
- Genetic algorithm based heuristics for the mapping problemComputers & Operations Research, 1995
- Heuristic technique for processor and link assignment in multicomputersIEEE Transactions on Computers, 1991
- Task allocation onto a hypercube by recursive mincut bipartitioningJournal of Parallel and Distributed Computing, 1990
- On the Mapping ProblemIEEE Transactions on Computers, 1981