Klaus Weide

Heterogeneity is the prevalent trend in the rapidly evolving high-performance computing (HPC) landscape in both hardware and application software. The diversity in hardware platforms, currently comprising various accelerators and a future possibility of specializable chiplets, poses a significant challenge for scientific software developers aiming to harness optimal performance across different computing platforms while maintaining the quality of solutions when their applications are simultaneously growing more complex. Code synthesis and code generation can provide mechanisms to mitigate this challenge. We have developed a divide and conquer approach where different aspects of performance are handled by different stand-alone tools that are interfaced with the application through generated code. This portability system, ORCHA, enables users to configure and orchestrate their computations among available resources on a platform by specifying a high-level recipe, thereby permitting a many-to-many paradigm where each recipe results in a different variant of the application. The core design goal is to let users decide the application’s hardware mapping and orchestration by editing only the high-level recipe—without modifying the maintained source code or binding the application to a particular runtime system. Tools in ORCHA distribution are: CG-Kit for translating the recipe into an execution graph; Milhoja to execute the graph by orchestrating data and task movement among hardware resources; and Macroprocessor that enables users to define their own code-shorthand for higher composability and easier management of code variants. Additionally, the design of ORCHA permits tools to work in a plug-and-play mode where the application can build and run without CG-Kit and Milhoja, and either tool can be swapped out for other tools with similar capabilities by modifying the code generation portion of ORCHA. In this paper, we describe the design of ORCHA and the role that code-generation plays in isolating applications from tools. We demonstrate the breadth of configurations ORCHA enables with a case study in which an application configuration is realized on three distinct hardware mappings—a GPU-centric, a CPU/GPU balanced, and a CPU/GPU concurrent layouts by using different recipes.

We present neutrino-transport algorithms implemented in the toolkit for high-order neutrino-radiation hydrodynamics ( thornado ) and their coupling to self-gravitating hydrodynamics within the adaptive mesh refinement–based multiphysics simulation framework F LASH -X. thornado , developed primarily for simulations of core-collapse supernovae (CCSNe), employs a spectral, six-species two-moment formulation with algebraic closure and special-relativistic observer corrections accurate to O ( v / c ) , and uses discontinuous Galerkin (DG) methods for phase-space discretization combined with implicit-explicit time stepping. A key development is a nonlinear neutrino–matter coupling algorithm based on nested fixed-point iteration with Anderson acceleration, enabling fully implicit treatment of collisional processes, including energy-coupling interactions such as neutrino–electron scattering and pair production. Coupling to finite-volume (FV) hydrodynamics is achieved through a hybrid DG-FV representation of the fluid variables and operator-split evolution within F LASH -X. The implementation is verified using basic transport tests with idealized opacities and relaxation and deleptonization problems with tabulated microphysics. Spherically symmetric CCSN simulations demonstrate accuracy and robustness of the coupled scheme, including close agreement with the CCSN simulation code Chimera . An axisymmetric CCSN simulation further demonstrates the viability of DG-based neutrino transport for multidimensional supernova modeling within F LASH -X. thornado ’s neutrino-transport solver is GPU-enabled using OpenMP offloading or OpenACC, and all CCSN applications included in this work use the GPU implementation. Together, these results establish a foundation for future enhancements in physics fidelity, numerical algorithms, and computational performance, for increasingly realistic large-scale CCSN simulations.

State-of-the-art multiphysics simulations running on large scale leadership computing platforms have many variables contributing to their performance and scaling behavior. We recently encountered an interesting performance anomaly in Flash-X, a multiphysics multicomponent simulation software, when characterizing its performance behavior on several large-scale HPC platforms. The anomaly was tracked down to the interaction between the use of dynamic allocation of scratch data and data locality in the cache hierarchy. In this paper we present the details of unexpected performance variability of Flash-X, its extensive analysis using the performance measurement tool TAU to collect the data and Python data analysis libraries to explore the data, and our insights from this experience. In this process, we discovered and removed or mitigated two additional performance limiting bottlenecks for performance tuning.

We present the newly incorporated gray radiation hydrodynamics capabilities of the FLASH code based on a radiation flux-limiter-aware hydrodynamics numerical implementation designed specifically for applications in astrophysical problems. The implemented numerical methods consist of changes in the unsplit hydrodynamics solver and adjustments in the flux-limited radiation diffusion unit. Our approach can handle problems in both the strong and weak radiation–matter coupling limits, as well as transitions between the two regimes. Appropriate extensions in the “Helmholtz” equation of state are implemented to treat two-temperature astrophysical plasmas involving the interaction between radiation and matter and the addition of a new opacity unit based on the OPAL opacity database, commonly used for astrophysical fluids. A set of radiation-hydrodynamics test problems is presented aiming to showcase the new capabilities of FLASH and to provide direct comparison to other similar software instruments available in the literature. To illustrate the capacity of FLASH to simulate phenomena occurring in stellar explosions, such as shock breakout, radiative precursors, and supernova ejecta heating due to the decays of radioactive ⁵⁶Ni and ⁵⁶Co, we also present 1D supernova simulations and compare the computed light curves to those of the SNEC code. The latest public release of FLASH with these enhanced capabilities is available for download and use by the broader astrophysics community.

The universe is permeated by magnetic fields, with strengths ranging from a femtogauss in the voids between the filaments of galaxy clusters to several teragauss in black holes and neutron stars. The standard model behind cosmological magnetic fields is the nonlinear amplification of seed fields via turbulent dynamo to the values observed. We have conceived experiments that aim to demonstrate and study the turbulent dynamo mechanism in the laboratory. Here, we describe the design of these experiments through simulation campaigns using FLASH, a highly capable radiation magnetohydrodynamics code that we have developed, and large-scale three-dimensional simulations on the Mira supercomputer at the Argonne National Laboratory. The simulation results indicate that the experimental platform may be capable of reaching a turbulent plasma state and determining the dynamo amplification. We validate and compare our numerical results with a small subset of experimental data using synthetic diagnostics.

The Biermann Battery effect is frequently invoked in cosmic magnetogenesis and studied in High-Energy Density laboratory physics experiments. Unfortunately, direct implementation of the Biermann effect in MHD codes is known to produce unphysical magnetic fields at shocks whose value does not converge with resolution. We show that this convergence breakdown is due to naive discretization, which fails to account for the fact that discretized irrotational vector fields have spurious solenoidal components that grow without bound near a discontinuity. We show that careful consideration of the kinetics of ion viscous shocks leads to a formulation of the Biermann effect that gives rise to a convergent algorithm. We note a novel physical effect a resistive magnetic precursor in which Biermann-generated field in the shock "leaks" resistively upstream. The effect appears to be potentially observable in experiments at laser facilities.