Manjunath Gorentla Venkata

Unified Collective Communication (UCC) is an API and library implementation of collective communication operations. The goal of UCC is to provide a unified API and library serving the collective communication needs of various workloads running on a wide variety of system architectures. Particularly, we aim to unify the collective communication interfaces and semantics and provide a common implementation framework for (i) parallel programming models, Deep Learning (DL) and I/O, (ii) collectives moving data in CPU main memory and device memory (GPU and Data Processing Unit (DPU)), and (iii) collective operations using software point-to-point and hardware transports. In this paper, we present an overview of UCC’s design, interfaces, semantics, and an implementation. We demonstrate UCC’s capabilities through evaluations with microbenchmarks representing diverse workloads and applications across various programming models including MPI, Partioned Global Address Space (PGAS) (OpenSHMEM), and PyTorch, on multiple hardware architectures (CPU, GPU and DPU).

The performance of OpenSHMEM applications is significantly influenced by the network message rate and the efficient utilization of bandwidth for small messages. While network hardware offers higher message rates, software overheads hinder the achievement of these rates within the programming model and applications. The absence of appropriate abstractions in the OpenSHMEM model exacerbates this challenge. For instance, message aggregation, which has been successful with MPI derived datatypes through user mode memory registration (UMR), encounters abstraction hurdles in the OpenSHMEM programming model. Although, such improvements can only be achieved using separate middleware built on top of OpenSHMEM and cannot be easily achieved natively. Furthermore, Quality of Service (QoS) for short messages under high loads is important for HPC applications, particularly in the case of OpenSHMEM. Congestion experienced in shared HPC systems due to contention for shared networks, I/O systems, etc., can create an adverse user experience and can impact overall system performance, leading to prolonged tail latencies. To address these issues, this paper introduces a Queues API abstraction for the OpenSHMEM programming model, enabling message aggregation and in-network prioritization for small data transfers. We present the OpenSHMEM API, build a prototype for OpenSHMEM queues, explore various message aggregation techniques, and evaluate NVIDIA InfiniBand virtual lanes for QoS in OpenSHMEM. Our results demonstrate a 2x-3x bandwidth improvement using OpenSHMEM queues with message aggregation. Additionally, we achieve a substantial reduction in tail latency, with improvements of up to 3.5x observed in congested scenarios for one-sided transfers.

Summary: The Exascale Computing Project (ECP) is currently the primary effort in the United States focused on developing “exascale” levels of computing capabilities, including hardware, software, and applications. In order to obtain a more thorough understanding of how the software projects under the ECP are using, and planning to use the Message Passing Interface (MPI), and help guide the work of our own project within the ECP, we created a survey. Of the 97 ECP projects active at the time the survey was distributed, we received 77 responses, 56 of which reported that their projects were using MPI. This paper reports the results of that survey for the benefit of the broader community of MPI developers.

The Exascale Computing Project (ECP) is currently the primary effort in the United States focused on developing “exascale” levels of computing capability, including hardware, software and applications. In order to obtain a more thorough understanding of how the software projects under the ECP are using, and planning to use the Message Passing Interface (MPI), and help guide the work of our own project within the ECP, we created a survey. Of the 97 ECP projects active at the time the survey was distributed, we received 77 responses, 56 of which reported that their projects were using MPI. This paper reports the results of that survey for the benefit of the broader community of MPI developers.

State-of-the-art synchronous graph processing frameworks face both inefficiency and imbalance issues that cause their performance to be suboptimal. These issues include the inefficiency of communication and the imbalanced graph computation/communication costs in an iteration. We propose to replace their conventional two-sided communication model with the one-sided counterpart. Accordingly, we design SHMEMGraph, an efficient and balanced graph processing framework that is formulated across a global memory space and takes advantage of the flexibility and efficiency of one-sided communication for graph processing. Through an efficient one-sided communication channel, SHMEMGraph utilizes the high-performance operations with RDMA while minimizing the resource contention within a computer node. In addition, SHMEMGraph synthesizes a number of optimizations to address both computation imbalance and communication imbalance. By using a graph of 1 billion edges, our evaluation shows that compared to the state-of-the-art Gemini framework, SHMEMGraph achieves an average improvement of 35.5% in terms of job completion time for five representative graph algorithms.

Extreme-scale applications (i.e., Big-Compute) are becoming increasingly data-intensive, i.e., producing and consuming increasingly large amounts of data. The HPC systems traditionally used for these applications are now used for Big-Data applications such as data analytics, social network analysis, machine learning, and genomics. As a consequence of these trends, the system architecture should be flexible and data-centric. This can already be witnessed in the pre-exascale systems with TBs of on-node hierarchical and heterogeneous memories, PBs of system memory, low-latency, high-throughput networks, and many threaded cores. As such, the pre-exascale systems suit the needs of both Big-Compute and Big-Data applications. Though the system architecture is flexible enough to support both Big-Compute and Big-Data, we argue there is a software gap. Particularly, we need data-centric abstractions to leverage the full potential of the system, i.e., there is a need for native support for data resilience, the ability to express data locality and affinity, mechanisms to reduce data movement, the ability to share data, and abstractions to express User's data usage and data access patterns. In this paper, we (i) show the need for taking a holistic approach towards data-centric abstractions, (ii) show how these approaches were realized in the SHARed data-structure centric Programming abstraction (SharP) library, a data-structure centric programming abstraction, and (iii) apply these approaches to a variety of applications that demonstrate its usefulness. Particularly, we apply these approaches to QMCPack and the Graph500 benchmark and demonstrate the advantages of this approach on extreme-scale systems.