Nikolaos D. Kallimanis

We present and evaluate the ExaNeSt prototype, which compactly packages 128 Xilinx ZU9EG MPSoCs, two TBytes of DRAM, and eight TBytes of SSD into a liquid-cooled rack, using a custom interconnection hardware based on 10 GB/s links. We developed this testbed in 2016–2019 in order to leverage the flexibility of FPGAs for experimenting with efficient hardware support for HPC communication among tens of thousands of processors and accelerators in the quest toward Exascale systems and beyond. In the years since then, we carefully studied this system, and we present our key design choices and insights resulting from our measurement and analysis. We developed this testbed, from architecture to the PCBs and the run-time software, within the ExaNeSt project. It is fully operational in configurations with up to 8 × 4 × 4 MPSoC nodes. It achieves high density through tight board design, while also leveraging state-of-the-art liquid cooling technology. In this article, we present a thorough architectural analysis, along with important aspects of our infrastructure development. Our custom interconnect includes a low-cost low-latency network interface, offering user-level, zero-copy RDMA, which we coupled with the ARMv8 processors in the MPSoCs. We further developed the corresponding runtimes that allow us to test real MPI applications on the large-scale testbed. We evaluated our platform through MPI microbenchmarks, mini application, and full MPI applications. Single-hop, one-way latency is 1.3 μs; approximately 0.47 μs out of these are attributed to network interface and the user-space library that exposes its functionality to the runtime. Latency over longer paths increases as expected, reaching 2.55 μs for a five-hop path. Bandwidth tests show that, for single-hop, link utilization reaches \(82\%\) of the theoretical capacity. Microbenchmarks based on MPI collectives reveal that broadcast latency scales as expected when the number of participating ranks increases. We also implemented a custom MPI_Allreduce accelerator in the network interface, which reduces the latency of such collectives by up to \(88\%\) . We assess performance scaling through weak and strong scaling tests for HPCG, LAMMPS, and the miniFE mini application; for all these tests, parallelization efficiency is at least \(69\%\) , or better.

In this work, we propose the $\lambda$-scanner snapshot, a variation of the snapshot object, which supports any fixed amount of $0 < \lambda \leq n$ different $SCAN$ operations being active at any given time. Whenever $\lambda$ is equal to the number of processes $n$ in the system, the $\lambda$-scanner object implements a multi-scanner object, while in case that $\lambda$ is equal to $1$, the $\lambda$-scanner object implements a single-scanner object. We present the $\lambda-Snap$ snapshot object, a wait-free $\lambda$-scanner snapshot implementation that has a step complexity of $O(\lambda)$ for $UPDATE$ operations and $O(\lambda m)$ for $SCAN$ operations. The space complexity of $\lambda-Snap$ is $O(\lambda m)$. $\lambda-Snap$ provides a trade-off between the step/space complexity and the maximum number of $SCAN$ operations that the system can afford to be active on any given point in time. The low space complexity that our implementations provide makes them more appealing in real system applications. Moreover, we provide a slightly modified version of the $\lambda-Snap$ implementation, which is called partial $\lambda-Snap$, that is able to support dynamic partial scan operations. In such an object, processes can execute modified $SCAN$ operations called $PARTIAL\_SCAN$ that could obtain a part of the snapshot object avoiding to read the whole set of components. In this work, we first provide a simple single-scanner version of $\lambda-Snap$, which is called $1-Snap$. We provide $1-Snap$ just for presentation purposes, since it is simpler than $\lambda-Snap$. The $UPDATE$ in $1-Snap$ has a step complexity of $O(1)$, while the $SCAN$ has a step complexity of $O(m)$. This implementation uses $O(m)$ $CAS$ registers.