Richard Tran Mills
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Richard Tran Mills has a total of 46 co-authors. Here are the top co-authors by number of shared publications:
- Jed Brown
University of Colorado Boulder
4 shared publications Jacob Faibussowitsch
Argonne National Laboratory, Lemont, IL, USA
3 shared publicationsJunchao Zhang
3 shared publications- Matthew Gregg Knepley
University at Buffalo, Buffalo, NY, USA
3 shared publications - Satish Balay
Argonne National Laboratory, Lemont, IL, USA
3 shared publications - Stefano Zampini
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
3 shared publications
Latest Publications
- 18 January 2025
- Published by SAGE Publications in The International Journal of High Performance Computing Applications
- Vol. 39 (2) , 306-325
PETSc/TAO developments for GPU-based early exascale systems
The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization via the Toolkit for Advanced Optimization (TAO). PETSc is used in dozens of scientific fields and is an important building block for many simulation codes. During the U.S. Department of Energy’s Exascale Computing Project, the PETSc team has made substantial efforts to enable efficient utilization of the massive fine-grain parallelism present within exascale compute nodes and to enable performance portability across exascale architectures. We recap some of the challenges that designers of numerical libraries face in such an endeavor, and then discuss the many developments we have made, which include the addition of new GPU backends, features supporting efficient on-device matrix assembly, better support for asynchronicity and GPU kernel concurrency, and new communication infrastructure. We evaluate the performance of these developments on some pre-exascale systems as well as the early exascale systems Frontier and Aurora, using compute kernel, communication layer, solver, and mini-application benchmark studies, and then close with a few observations drawn from our experiences on the tension between portable performance and other goals of numerical libraries.
- 18 January 2025
- Published by SAGE Publications in The International Journal of High Performance Computing Applications
- Vol. 39 (2) , 306-325
PETSc/TAO developments for GPU-based early exascale systems
The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization via the Toolkit for Advanced Optimization (TAO). PETSc is used in dozens of scientific fields and is an important building block for many simulation codes. During the U.S. Department of Energy’s Exascale Computing Project, the PETSc team has made substantial efforts to enable efficient utilization of the massive fine-grain parallelism present within exascale compute nodes and to enable performance portability across exascale architectures. We recap some of the challenges that designers of numerical libraries face in such an endeavor, and then discuss the many developments we have made, which include the addition of new GPU backends, features supporting efficient on-device matrix assembly, better support for asynchronicity and GPU kernel concurrency, and new communication infrastructure. We evaluate the performance of these developments on some pre-exascale systems as well as the early exascale systems Frontier and Aurora, using compute kernel, communication layer, solver, and mini-application benchmark studies, and then close with a few observations drawn from our experiences on the tension between portable performance and other goals of numerical libraries.
- Open Access
- 13 December 2024
- Published by Copernicus GmbH
Enabling End-to-End Accelerated Multiphysics Simulations in the Exascale Era Using PETSc
The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization; it is used in dozens of scientific fields, and has been an important building block for many computational geoscience applications. Starting from the terascale era in the 1990s and continuing into the present dawn of the exascale era, a major goal of PETSc development has been achieving the scalability required to fully utilize leadership-class supercomputers. We will describe some of the algorithmic developments made during the era in which achieving inter-node scalability was the primary challenge to enabling extreme-scale computation, and then survey the challenges posed by the current era in which harnessing the abundant fine-scale parallelism within compute nodes -- primarily in the form of GPU-based accelerators -- has assumed at least equal importance. We will discuss how the PETSc design for performance portability addresses these challenges while stressing flexibility and extensibility by separating the programming model used by application code from that used by the library. Additionally, we will discuss recent developments in PETSc's communication module, PetscSF, that enable flexibility and scalable performance across large GPU-based systems while overcoming some of the difficulties posed by working directly with the Message Passing Interface (MPI) on such systems. A particular goal of this talk will be to go beyond simply describing the work performed to prepare PETSc and simulation codes that rely on it to run on exascale-class systems, but to enumerate the challenges we encountered and to share the essential lessons learned that can help other developers to prepare and optimize their high-performance scientific computing codes for the exascale era.
- 13 December 2024
- Published by Copernicus GmbH
Open AccessEnabling End-to-End Accelerated Multiphysics Simulations in the Exascale Era Using PETSc
The Portable Extensible Toolkit for Scientific Computation (PETSc) library provides scalable solvers for nonlinear time-dependent differential and algebraic equations and for numerical optimization; it is used in dozens of scientific fields, and has been an important building block for many computational geoscience applications. Starting from the terascale era in the 1990s and continuing into the present dawn of the exascale era, a major goal of PETSc development has been achieving the scalability required to fully utilize leadership-class supercomputers. We will describe some of the algorithmic developments made during the era in which achieving inter-node scalability was the primary challenge to enabling extreme-scale computation, and then survey the challenges posed by the current era in which harnessing the abundant fine-scale parallelism within compute nodes -- primarily in the form of GPU-based accelerators -- has assumed at least equal importance. We will discuss how the PETSc design for performance portability addresses these challenges while stressing flexibility and extensibility by separating the programming model used by application code from that used by the library. Additionally, we will discuss recent developments in PETSc's communication module, PetscSF, that enable flexibility and scalable performance across large GPU-based systems while overcoming some of the difficulties posed by working directly with the Message Passing Interface (MPI) on such systems. A particular goal of this talk will be to go beyond simply describing the work performed to prepare PETSc and simulation codes that rely on it to run on exascale-class systems, but to enumerate the challenges we encountered and to share the essential lessons learned that can help other developers to prepare and optimize their high-performance scientific computing codes for the exascale era.
- 19 June 2024
- Published by Elsevier in Journal of Computational Science
- Vol. 81
Development of a multi-layer canopy model for E3SM Land Model with support for heterogeneous computing
- 19 June 2024
- Published by Elsevier in Journal of Computational Science
- Vol. 81
Development of a multi-layer canopy model for E3SM Land Model with support for heterogeneous computing
- Open Access
- 16 April 2023
- Published by Wiley
Development of a multi-layer canopy model for E3SM Land Model with support for heterogeneous computing
The vertical structure of vegetation canopies creates micro-climates, which can substantially affect ecosystem responses to climate change. However, the land components of most Earth System Models, including the Energy Exascale Earth System Model (E3SM), typically neglect vertical canopy structure by using a single layer big-leaf representation to simulate water, \cotwo, and energy exchanges between the land and the atmosphere. In this study, we developed a standalone Multi-Layer Canopy Model (MLCMv1) for the E3SM Land Model (ELM) to resolve the micro-climate created by vegetation canopies. The support for the heterogeneous computation architectures is included by using the Portable Extensible Toolkit for Scientific Programming. The numerical implementation of ELM-MLCMv1 was verified against CLM-ml\_v1 for a month-long simulation using data from the Ameriflux US-University of Michigan Biological Station (US-UMB) site. Model structural uncertainty was explored by performing control simulations for five stomatal conductance models (SCMs). All SCMs after calibration were able to accurately match observations of sensible and latent heat flux, though the bias of the three SCMs with plant hydrodynamics (PHD) was slightly lower than that of two SCMs without PHD. Additionally, six idealized simulations were performed to study the impact of environmental variables on canopy processes. All SCMs agreed on the direction of simulated changes in canopy processes due to the changes in these environmental variables. ELM-MLCMv1 achieves a speedup of 25-50 times when comparing performance on a GPU relative to a CPU. This study provides the first necessary model development for including the representation of vertical canopies within ELM.
- 16 April 2023
- Published by Wiley
Open AccessDevelopment of a multi-layer canopy model for E3SM Land Model with support for heterogeneous computing
The vertical structure of vegetation canopies creates micro-climates, which can substantially affect ecosystem responses to climate change. However, the land components of most Earth System Models, including the Energy Exascale Earth System Model (E3SM), typically neglect vertical canopy structure by using a single layer big-leaf representation to simulate water, \cotwo, and energy exchanges between the land and the atmosphere. In this study, we developed a standalone Multi-Layer Canopy Model (MLCMv1) for the E3SM Land Model (ELM) to resolve the micro-climate created by vegetation canopies. The support for the heterogeneous computation architectures is included by using the Portable Extensible Toolkit for Scientific Programming. The numerical implementation of ELM-MLCMv1 was verified against CLM-ml\_v1 for a month-long simulation using data from the Ameriflux US-University of Michigan Biological Station (US-UMB) site. Model structural uncertainty was explored by performing control simulations for five stomatal conductance models (SCMs). All SCMs after calibration were able to accurately match observations of sensible and latent heat flux, though the bias of the three SCMs with plant hydrodynamics (PHD) was slightly lower than that of two SCMs without PHD. Additionally, six idealized simulations were performed to study the impact of environmental variables on canopy processes. All SCMs agreed on the direction of simulated changes in canopy processes due to the changes in these environmental variables. ELM-MLCMv1 achieves a speedup of 25-50 times when comparing performance on a GPU relative to a CPU. This study provides the first necessary model development for including the representation of vertical canopies within ELM.
- Open Access
- 15 November 2022
- Published by The Open Journal in Journal of Open Source Software
- Vol. 7 (79) , 4531
DMStag: Staggered, Structured Grids for PETSc
"Staggered grid" codes arise in many applications, where a physical problem is discretized and solved in terms of unknowns located at different locations on a structured grid. For example, the Navier-Stokes and Stokes equations can be solved on a staggered grid with the classical Marker and Cell (MAC) method (Harlow & Welch, 1965), storing pressure unknowns in element centers and velocities on element boundaries. This gives a simple and compact method which exhibits superior numerical properties to colocated grid methods which consider all unknowns to be located on the same grid of points.
- 15 November 2022
- Published by The Open Journal in Journal of Open Source Software
- Vol. 7 (79) , 4531
Open AccessDMStag: Staggered, Structured Grids for PETSc
"Staggered grid" codes arise in many applications, where a physical problem is discretized and solved in terms of unknowns located at different locations on a structured grid. For example, the Navier-Stokes and Stokes equations can be solved on a staggered grid with the classical Marker and Cell (MAC) method (Harlow & Welch, 1965), storing pressure unknowns in element centers and velocities on element boundaries. This gives a simple and compact method which exhibits superior numerical properties to colocated grid methods which consider all unknowns to be located on the same grid of points.
- Open Access
- 11 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Computing in Science & Engineering
- Vol. 24 (3) , 6-15
The PETSc Community as Infrastructure
The communities who develop and support open source scientific software packages are crucial to the utility and success of such packages. Moreover, these communities form an important part of the human infrastructure that enables scientific progress. This paper discusses aspects of the PETSc (Portable Extensible Toolkit for Scientific Computation) community, its organization, and technical approaches that enable community members to help each other efficiently and effectively.
- 11 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Computing in Science & Engineering
- Vol. 24 (3) , 6-15
Open AccessThe PETSc Community as Infrastructure
The communities who develop and support open source scientific software packages are crucial to the utility and success of such packages. Moreover, these communities form an important part of the human infrastructure that enables scientific progress. This paper discusses aspects of the PETSc (Portable Extensible Toolkit for Scientific Computation) community, its organization, and technical approaches that enable community members to help each other efficiently and effectively.
- Open Access
- 28 February 2022
- Published by Wiley in ESS Open Archive
Wildfire Classification using PETSc-based Support Vector Machines on Distributed-Memory GPU-based Parallel Computers
As high-resolution geospatiotemporal data sets from observatory networks, remote sensing platforms, and computational Earth systems increase in abundance, fidelity, and richness, machine learning approaches that can fully utilize increasingly powerful parallel computing resources are becoming essential for analysis and exploration of such data sets. We explore one such approach, applying a state-of-the-art distributed memory parallel implementation of Support Vector Machine (SVM) classification to large remote-sensing data sets. We have used MODIS 8-day surface reflectance (MOD09A1) and land surface temperature (MOD11A2) for classifying wildfires over Alaska and California. Monitoring Trends in Burn Severity (MTBS) burn perimeter data was used to set boundaries of burned and unburned areas for our two-class problem. MTBS covers years from 1984-2019, recording only fires over 1000 acres or greater in the western United States. We seek to find a parallel computing solution (using the PermonSVM solver, described below) to accurately classify wildfires and find smaller unrecorded wildfires. An initial assessment for wildfire classification over interior Alaska shows that PermonSVM has an accuracy of 96% and over 5000 false positives (i.e., fires unrecorded in MTBS). Next steps include mapping larger regions over Alaska and California and understanding the tradeoffs of scalability and accuracy. The parallel tool we employ is PermonSVM, which is built on top of the widely-used open source toolkit PETSc, the Portable, Extensible Toolkit for Scientific Computation. Recent developments in PETSc have focused on supporting cutting-edge GPU-based high-performance computing (HPC) architectures, and these can be easily leveraged in PermonSVM by using appropriate GPU-enabled matrix and vector types in PETSc. We achieve significant GPU speedup for the SVM calculations on the Summit supercomputer at Oak Ridge National Laboratory – currently one of the best available “at scale” proxies for upcoming exascale-class supercomputers – and are actively working to further improve computational efficiency on Summit as well as on prototype exascale node architectures.
- 28 February 2022
- Published by Wiley in ESS Open Archive
Open AccessWildfire Classification using PETSc-based Support Vector Machines on Distributed-Memory GPU-based Parallel Computers
As high-resolution geospatiotemporal data sets from observatory networks, remote sensing platforms, and computational Earth systems increase in abundance, fidelity, and richness, machine learning approaches that can fully utilize increasingly powerful parallel computing resources are becoming essential for analysis and exploration of such data sets. We explore one such approach, applying a state-of-the-art distributed memory parallel implementation of Support Vector Machine (SVM) classification to large remote-sensing data sets. We have used MODIS 8-day surface reflectance (MOD09A1) and land surface temperature (MOD11A2) for classifying wildfires over Alaska and California. Monitoring Trends in Burn Severity (MTBS) burn perimeter data was used to set boundaries of burned and unburned areas for our two-class problem. MTBS covers years from 1984-2019, recording only fires over 1000 acres or greater in the western United States. We seek to find a parallel computing solution (using the PermonSVM solver, described below) to accurately classify wildfires and find smaller unrecorded wildfires. An initial assessment for wildfire classification over interior Alaska shows that PermonSVM has an accuracy of 96% and over 5000 false positives (i.e., fires unrecorded in MTBS). Next steps include mapping larger regions over Alaska and California and understanding the tradeoffs of scalability and accuracy. The parallel tool we employ is PermonSVM, which is built on top of the widely-used open source toolkit PETSc, the Portable, Extensible Toolkit for Scientific Computation. Recent developments in PETSc have focused on supporting cutting-edge GPU-based high-performance computing (HPC) architectures, and these can be easily leveraged in PermonSVM by using appropriate GPU-enabled matrix and vector types in PETSc. We achieve significant GPU speedup for the SVM calculations on the Summit supercomputer at Oak Ridge National Laboratory – currently one of the best available “at scale” proxies for upcoming exascale-class supercomputers – and are actively working to further improve computational efficiency on Summit as well as on prototype exascale node architectures.
- 1 December 2021
- Published by Elsevier in Parallel Computing
- Vol. 108
Toward performance-portable PETSc for GPU-based exascale systems
- 1 December 2021
- Published by Elsevier in Parallel Computing
- Vol. 108
Toward performance-portable PETSc for GPU-based exascale systems
- 26 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (4) , 842-853
The PetscSF Scalable Communication Layer
PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. In this article, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.
- 26 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (4) , 842-853
The PetscSF Scalable Communication Layer
PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. In this article, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.