Efficiency and scalability of fully-resolved fluid-particle simulations on heterogeneous CPU-GPU architectures
Open Access
- 10 January 2025
- journal article
- research article
- Published by SAGE Publications in The International Journal of High Performance Computing Applications
- Vol. 39 (3) , 345-363
Abstract
Current supercomputers often have a heterogeneous architecture using both conventional Central Processing Units (CPUs) and Graphics Processing Units (GPUs). At the same time, numerical simulation tasks frequently involve multiphysics scenarios whose components run on different hardware due to multiple reasons, e.g., architectural requirements, pragmatism, etc. This leads naturally to a software design where different simulation modules are mapped to different subsystems of the heterogeneous architecture. We present a detailed performance analysis for such a hybrid four-way coupled simulation of a fully resolved particle-laden flow. The Eulerian representation of the flow utilizes GPUs, while the Lagrangian model for the particles runs on conventional CPUs. Two characteristic model situations involving dense and dilute particle systems are used as benchmark scenarios. First, a roofline model is employed to predict the node level performance and to show that the lattice-Boltzmann-based Eulerian fluid simulation reaches very good performance on a single GPU. Furthermore, the GPU-GPU communication for a large-scale Eulerian flow simulation results in only moderate slowdowns. This is due to the efficiency of the CUDA-aware MPI communication, combined with the use of communication hiding techniques. On 1024 A100 GPUs, an overall parallel efficiency of up to 71% is achieved. While the flow simulation has good performance characteristics, the integration of the stiff Lagrangian particle system requires frequent CPU-CPU communications that can become a bottleneck, especially when simulating the dense particle system. Additionally, special attention is paid to the CPU-GPU communication overhead since this is essential for coupling the particles to the flow simulation. However, thanks to our problem-aware co-partitioning, the CPU-GPU communication overhead is found to be negligible. As a lesson learned from this development, four criteria are postulated that a hybrid implementation must meet for the efficient use of heterogeneous supercomputers.Keywords
All Related Versions
Funding Information
- European High Performance Computing Joint Undertaking (101093393)
- Deutsche Forschungsgemeinschaft (433735254)
This publication has 46 references indexed in Scilit:
- waLBerla: A block-structured high-performance framework for multiphysics simulationsComputers & Mathematics with Applications, 2021
- A CPU-GPU cross-platform coupled CFD-DEM approach for complex particle-fluid flowsChemical Engineering Science, 2020
- Improved coupling of time integration and hydrodynamic interaction in particle suspensions using the lattice Boltzmann and discrete element methodsComputers & Mathematics with Applications, 2018
- Fully Resolved Simulations of Dune Formation in Riverbeds2017
- The Lattice Boltzmann Method2017
- Fluid–particle interaction in turbulent open channel flow with fully-resolved mobile bedsAdvances in Water Resources, 2014
- Large-scale CFD–DEM simulations of fluidized granular systemsChemical Engineering Science, 2013
- Discrete particle simulation of gas–solid two-phase flows with multi-scale CPU–GPU hybrid computationChemical Engineering Journal, 2012
- LBM based flow simulation using GPU computing processorComputers & Mathematics with Applications, 2010
- Massively parallel granular flow simulations with non-spherical particlesComputer Science - Research and Development, 2010