Flavio Cesar Cunha Galeazzo

In this work, the authors focus on assessing the impact of the AMD EPYC processor architecture on the performance of CFD applications. Several generations of architectures were analyzed, such as Rome, Milan, Milan X, Genoa, Genoa X and Bergamo, characterized by a different number of cores (64-128), L3 cache size (256 - 1152 MB) and RAM type (8-channel DDR4 or 12-channel DDR5). The research was conducted based on the OpenFOAM application using two memory-bound models: motorBike and Urban Air Pollution. In order to compare the performance of applications on different architectures, the FVOPS (Finite VOlumes solved Per Second) metric was introduced, which allows a direct comparison of the performance on the different architectures. It was noticed that local maximum performance occurs in the grid sizes assigned to the processing process, which is related to individual processor attributes. Additionally, the behavior of the models was analyzed in detail using the software profiling analysis tool AMD uProf to reveal the applications' interaction with the hardware. It enabled fine-tuned monitoring of the CPU's behaviours and identified potential inefficiencies in AMD EPYC CPUs. Particular attention was paid to the effective use of L2 and L3 cache memory in the context of their capacity and the bandwidth of memory channels, which are a key factor in memory-bound applications. Processor features were analyzed from a cross-platform perspective, which allowed for the determination of metrics of particular importance in terms of their impact on the performance achieved by CFD applications.

OpenFOAM is a CFD software widely used in both industry and academia. The exaFOAM project aims at enhancing the HPC scalability of OpenFOAM, while identifying its current bottlenecks and proposing ways to overcome them. For the assessment of the software components and the code profiling during the code development, lightweight but significant benchmarks should be used. The answer was to develop microbenchmarks, with a small memory footprint and short runtime. The name microbenchmark does not mean that they have been prepared to be the smallest possible test cases, as they have been developed to fit in a compute node, which usually has dozens of compute cores. The microbenchmarks cover a broad band of applications: incompressible and compressible flow, combustion, viscoelastic flow and adjoint optimization. All benchmarks are part of the OpenFOAM HPC Technical Committee repository and are fully accessible. The performance using HPC systems with Intel and AMD processors (x86_64 architecture) and Arm processors (aarch64 architecture) have been benchmarked. For the workloads in this study, the mean performance with the AMD CPU is 62% higher than with Arm and 42% higher than with Intel. The AMD processor seems particularly suited resulting in an overall shorter time-to-solution.

This paper presents a new mesh data layout for parallel I/O of linear unstructured polyhedral meshes. The new mesh representation infers coherence across entities of different topological dimensions, i.e., grid cells, faces, and points. The coherence due to cell-to-face and face-to-point connectivities of the mesh is formulated as a tree data structure distributed across processors. The mesh distribution across processors creates consecutive and contiguous slices that render an optimized data access pattern for parallel I/O. A file format using the coherent mesh representation, developed and tested with OpenFOAM, enables the usability of the software at unprecedented scales. Further implications of the coherent and sliceable mesh representation arise due to simplifications in partitioning and diminished pre- and post-processing overheads.

The performance of OpenFOAM in strong scaling tests on HPC systems with AMD EPYC processors exhibits a pronounced superlinear speedup. Simple test cases show superlinear speedups of over 300%, which significantly impacts the efficient use of computing resources. With the last generation of HPC architectures, a superlinear speedup of about 10% to 20% was well expected and accepted by CFD users. The measured superlinear speedup is much more pronounced and withstands the communication overhead to even larger scales. A detailed performance analysis of OpenFOAM follows, employing various High-Performance Computing (HPC) architectures, including AMD, ARM and Intel systems. The performance metric FVOPS (Finite VOlumes solved Per Second) is introduced to compare the performance of Computational Fluid Dynamics (CFD) applications when varying the grid size, as occurs in a strong scaling test. The achievable FVOPS depends on various factors, including the simulation type, boundary conditions, and especially the grid size of a use case. Analysing FVOPS on a single node level with varying grid size shows a significant difference in performance and cache utilization, which explains the large superlinear speedups seen in the strong scaling tests. Furthermore, FVOPS can be used as a simple benchmark to determine the optimal number of grid elements per rank to simulate a given use case at peak efficiency on a given platform, resulting in time, energy, and cost savings. The FVOPS metric also facilitates the direct comparison between different HPC architectures. The tests using AMD, ARM, and Intel processors show a peak in performance when employing around 10,000 grid elements per core. The presence of a large L3 cache on AMD processors is particularly advantageous, as indicated by L3 cache miss rates observed on AMD EPYC processors. Our results suggest that future HPC architectures with larger caches and higher memory bandwidth would benefit the CFD community.

OpenFOAM is a CFD software widely used in both industry and academia. The exaFOAM project aims at enhancing the HPC scalability of OpenFOAM, while identifying its current bottlenecks and proposing ways to overcome them. For the assessment of the software components and the code profiling during the code development, lightweight but significant benchmarks should be used. The answer was to develop microbenchmarks, with a small memory footprint and short runtime. The name microbenchmark does not mean that they have been prepared to be the smallest possible test cases, as they have been developed to fit in a compute node, which usually has dozens of compute cores. The microbenchmarks cover a broad band of applications: incompressible and compressible flow, combustion, viscoelastic flow and adjoint optimisation. All benchmarks are part of the OpenFOAM HPC Technical Committee repository and are fully accessible. The performance using HPC systems with Intel and AMD processors (x86_64 architecture) and Arm processors (aarch64 architecture) have been benchmarked. For the workloads in this study, the AMD processor seems particularly suited resulting in an overall shorter time-to-solution.

We evaluate the accuracy of the actuator line model (ALM) approach by performing simulations for the NREL 5‐MW wind turbine in uniform inflow using three large eddy simulation codes. The power and thrust coefficients obtained using the three codes agree within 1% when the grid spacing m and are cross‐validated against blade element momentum (BEM) theory. We find that the results of ALM converge towards BEM theory without the need for tip correction when the numerical resolution is increased. For m, the difference between the power and thrust coefficient obtained using ALM and BEM is 4.5% and 2.1%, respectively, although we note that no absolute convergence between ALM and BEM can be obtained as both models use different assumptions, such as the use of a force projection method in the ALM. The difference in the local axial and tangential forces along the blades obtained from ALM simulations using m and m can be as large as 10%. The effect of the number of actuator points on the obtained turbine power and thrust coefficients is limited as the results converge when the spacing between the actuator points is about three times the grid spacing. This insight on the required number of blade points can be used to improve the efficiency of actuator line simulations.