Strong scaling for numerical weather prediction at petascale with the atmospheric model NUMA
Authors:
Open Access
- 5 April 2018
- journal article
- research article
- Published by SAGE Publications in The International Journal of High Performance Computing Applications
- Vol. 33 (2) , 411-426
Abstract
Numerical weather prediction (NWP) has proven to be computationally challenging due to its inherent multiscale nature. Currently, the highest resolution global NWP models use a horizontal resolution of 9 km. At this resolution, many important processes in the atmosphere are not resolved. Needless to say, this introduces errors. In order to increase the resolution of NWP models, highly scalable atmospheric models are needed. The non-hydrostatic unified model of the atmosphere (NUMA), developed by the authors at the Naval Postgraduate School, was designed to achieve this purpose. NUMA is used by the Naval Research Laboratory, Monterey as the engine inside its next generation weather prediction system NEPTUNE. NUMA solves the fully compressible Navier–Stokes equations by means of high-order Galerkin methods (both spectral element as well as discontinuous Galerkin methods can be used). NUMA is capable of running middle and upper atmosphere simulations since it does not make use of the shallow-atmosphere approximation. This article presents the performance analysis and optimization of the spectral element version of NUMA. The performance at different optimization stages is analyzed using a theoretical performance model as well as measurements via hardware counters. Machine-independent optimization is compared to machine-specific optimization using Blue Gene (BG)/Q vector intrinsics. The best portable version of the main computations was found to be about two times slower than the best non-portable version. By using vector intrinsics, the main computations reach 1.2 PFlops on the entire IBM Blue Gene supercomputer Mira (12% of the theoretical peak performance). The article also presents scalability studies for two idealized test cases that are relevant for NWP applications. The atmospheric model NUMA delivers an excellent strong scaling efficiency of 99% on the entire supercomputer Mira using a mesh with 1.8 billion grid points. This allows running a global forecast of a baroclinic wave test case at a 3-km uniform horizontal resolution and double precision within the time frame required for operational weather prediction.Keywords
All Related Versions
Funding Information
- Air Force Office of Scientific Research (Computational Mathematics program)
This publication has 25 references indexed in Scilit:
- Comparison between adaptive and uniform discontinuous Galerkin simulations in dry 2D bubble experimentsJournal of Computational Physics, 2013
- Implicit-Explicit Formulations of a Three-Dimensional Nonhydrostatic Unified Model of the Atmosphere (NUMA)SIAM Journal on Scientific Computing, 2013
- Continuous and discontinuous Galerkin methods for a scalable three-dimensional nonhydrostatic atmospheric model: Limited-area modeJournal of Computational Physics, 2012
- Parallel Implementation and Scalability of Cloud Resolving EULAG ModelLecture Notes in Computer Science, 2012
- CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere ModelThe International Journal of High Performance Computing Applications, 2011
- p4est: Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of OctreesSIAM Journal on Scientific Computing, 2011
- Computational aspects of a scalable high-order discontinuous Galerkin atmospheric dynamical coreComputers & Fluids, 2009
- A study of spectral element and discontinuous Galerkin methods for the Navier–Stokes equations in nonhydrostatic mesoscale atmospheric modeling: Equation sets and test casesJournal of Computational Physics, 2008
- A baroclinic instability test case for atmospheric model dynamical coresQuarterly Journal of the Royal Meteorological Society, 2006
- On the Quality of Partitions Based on Space-Filling Curves2002