An Architecture-Agnostic High-Order Discontinuous Galerkin Framework for Compressible Flows

Abstract

With the recent proliferation of heterogeneous, GPU-accelerated supercomputers, high-order computational fluid dynamics (CFD) simulations of complex, turbulent flows are more accessible than ever. To leverage the computing power of these machines, CFD software must adapt. However, complicating the situation is the emerging need to support hardware from multiple GPU vendors. Addressing this need is the GPU-accelerated, discontinuous Galerkin spectral element method (DGSEM) framework GALÆXI, a high-order, open source, architecture-agnostic toolchain for the study of complex, compressible, turbulent flows on unstructured, hexahedral grids. GPU-accelerated computations with GALÆXI are possible on GPU hardware by interfacing Fortran source code to the vendor models CUDA C++ for NVIDIA and HIP C++ for AMD. The DGSEM implementation in GALÆXI was verified using the method of manufactured solutions to rigorously confirm the expected order of convergence. Simulations of a compressible Taylor-Green-Vortex also demonstrated excellent agreement with reference solutions across all supported architectures. GALÆXI achieved near ideal strong and weak scaling on GPU hardware from both NVIDIA and AMD. In the largest case, GALÆXI performed a simulation with 67.1 billion degrees of freedom on 65,536 AMD MI250X graphics compute devices with a parallel efficiency of 82.6%. Comparing node-to-node performance, GPU simulations offered speedups between 7.75x and 8.08x over CPU computations in time-to-solution while consuming less than half the energy. To demonstrate GALÆXI's effectiveness for production-scale simulations, wall-resolved large eddy simulations of the transonic flow past a NACA 64A-110 airfoil and an ONERA OAT15A airfoil under shock buffet conditions were computed.