NUMA on Manycore Architectures

The Nonhydrostatic Unified Model of the Atmosphere (NUMA) has been retrofitted in order to be able to run on Manycore computers including Nvidia GPUs as well as Intel's Xeon Phis (Knights Landing). NUMA is developed in my group in the Department of Applied Mathematics of the Naval Postgraduate School and is the engine that runs inside the U.S. Navy's next-generation numerical weather prediction model (called NEPTUNE=Navy's Environmental Prediction sysTem Using the NUMA corE). The main advantage of the approach we have taken with NUMA is that, using the same code-base, NUMA is able to run across quite different computer hardware including: standard x86 CPUs, GPUs, Xeon Phi (KNL), and possibly even other types of co-processors/accelerators. In our recent paper (https://www.researchgate.net/publication/309399443_Acceleration_of_the_Implicit-Explicit_Non-hydrostatic_Unified_Model_of_the_Atmosphere_NUMA_on_Manycore_Processors) we show that NUMA approaches the NOAA time-frame constraint by performing a 24-hour forecast (dry dynamics) within 5 wallclock minutes at a 3 km global grid resolution (red curves in the figure) while only using 4096 GPUs of the Titan supercomputer (for an acoustic wave problem on the sphere). This result was achieved thanks to our porting of the ENTIRE NUMA code-base onto the GPU including all of the dynamics, implicit time-integrators, iterative solvers, and preconditioners. The figure shows the wallclock time required for a 24-hour forecast (vertical axis) as a function of the number of GPUs (horizontal axis) for explicit time-integration (RK), 1D-IMEX, as well as full 3D-IMEX.