MPI Allgather Utilizing CXL Shared Memory Pool in Multi-Node Computing Systems
Authors:
- 15 December 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 332-337
Abstract
In Artificial Intelligence (AI) and high-performance computing (HPC), growing data and model sizes require distributed processing across multiple nodes due to single-node limitations, increasing inter-node communication. To address these challenges, we propose a novel MPI allgather method leveraging CXL technology, which supports composable architectures and dynamic resource allocation in data centers and HPC systems. Notably, CXL 3.1 facilitates cache coherence among nodes. The proposed allgather method uses the CXL shared memory pool as a communication buffer, outperforming existing algorithms for two reasons: First, CXL provides lower latency than Ethernet and IB, and second, by using the CXL shared memory pool as a shared communication buffer across multiple nodes, it significantly reduces the number of communications. To the best of our knowledge, this work is the first to explore combining MPI collective communication with CXL technology to optimize MPI allgather. Our proposed allgather method significantly reduces communication latency compared to traditional allgather methods by up to 42.14x, with a minimum improvement of 2.91x, as measured using the OSU Micro-Benchmark (OMB), a standard MPI benchmarking suite.Keywords
Funding Information
- National Research Foundation
This publication has 14 references indexed in Scilit:
- Research on the CXL Memory2024
- Optimizing MPI Collectives on Shared Memory Multi-Cores2023
- CXL over Ethernet: A Novel FPGA-based Memory Disaggregation Design in Data Centers2023
- Pond: CXL-Based Memory Pooling Systems for Cloud Platforms2023
- Memory Pooling With CXLIEEE Micro, 2023
- Evaluating Emerging CXL-enabled Memory Pooling for HPC Systems2022
- Efficient sparse collective communication and its application to accelerate distributed deep learning2021
- Synthesizing optimal collective algorithms2021
- Framework for Scalable Intra-Node Collective Operations using Shared Memory2018
- SHMEMPMI -- Shared Memory Based PMI for Improved Performance and Scalability2016