Scaling Computational Fluid Dynamics: In Situ Visualization of NekRS using SENSEI
Authors:
- Victor A. MateevitsiVMVictor A. Mateevitsi
- Mathis BodeMBMathis Bode
- Nicola FerrierNFNicola Ferrier
- Paul FischerPFPaul Fischer
- Jens Henrik GöbbertJGJens Henrik Göbbert
- Joseph A. InsleyJIJoseph A. Insley
- Yu-Hsiang LanYLYu-Hsiang Lan
- Misun MinMMMisun Min
- Michael E. PapkaMPMichael E. Papka
- Saumil PatelSPSaumil Patel
- Silvio RizziSRSilvio Rizzi
- Jonathan WindgassenJWJonathan Windgassen
Preprint
Abstract
In the realm of Computational Fluid Dynamics (CFD), the demand for memory and computation resources is extreme, necessitating the use of leadership-scale computing platforms for practical domain sizes. This intensive requirement renders traditional checkpointing methods ineffective due to the significant slowdown in simulations while saving state data to disk. As we progress towards exascale and GPU-driven High-Performance Computing (HPC) and confront larger problem sizes, the choice becomes increasingly stark: to compromise data fidelity or to reduce resolution. To navigate this challenge, this study advocates for the use of in situ analysis and visualization techniques. These allow more frequent data "snapshots" to be taken directly from memory, thus avoiding the need for disruptive checkpointing. We detail our approach of instrumenting NekRS, a GPU-focused thermal-fluid simulation code employing the spectral element method (SEM), and describe varied in situ and in transit strategies for data rendering. Additionally, we provide concrete scientific use-cases and report on runs performed on Polaris, Argonne Leadership Computing Facility's (ALCF) 44 Petaflop supercomputer and Jülich Wizard for European Leadership Science (JUWELS) Booster, Jülich Supercomputing Centre's (JSC) 71 Petaflop High Performance Computing (HPC) system, offering practical insight into the implications of our methodology.Keywords
All Related Versions
- Version 1, 2023-12-15, ArXiv
- Version 2, 2023-12-18, ArXiv
- Published version: , 862.
This publication has 13 references indexed in Scilit:
- NekRS, a GPU-accelerated spectral element Navier–Stokes solverParallel Computing, 2022
- Convective mesoscale turbulence at very low Prandtl numbersJournal of Fluid Mechanics, 2022
- Ascent: A Flyweight In Situ Library for Exascale Simulations2022
- In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView CatalystThe Journal of Supercomputing, 2021
- A GPU accelerated discontinuous Galerkin incompressible flow solverJournal of Computational Physics, 2019
- The Exascale Computing ProjectComputing in Science & Engineering, 2017
- The SENSEI Generic In Situ Interface2016
- OSPRay - A CPU Ray Tracing Framework for Scientific VisualizationIEEE Transactions on Visualization and Computer Graphics, 2016
- VTK-m: Accelerating the Visualization Toolkit for Massively Threaded ArchitecturesIEEE Computer Graphics and Applications, 2016
- In-situ processing and visualization for ultrascale simulationsJournal of Physics: Conference Series, 2007