Method for portable, scalable, and performant GPU-accelerated simulation of multiphase compressible flow

Anand Radhakrishnan; Henry Le Berre; Benjamin Wilfong; Jean-Sebastien Spratt; Mauro Rodriguez Jr.; Tim Colonius; Spencer H. Bryngelson

doi:10.1016/j.cpc.2024.109238

arXiv · 2305.09163

Method for portable, scalable, and performant GPU-accelerated simulation of multiphase compressible flow

Anand Radhakrishnan·Henry Le Berre·Benjamin Wilfong·Jean-Sebastien Spratt·Mauro Rodriguez Jr.·Tim Colonius·Spencer H. Bryngelson

View original source ↗Read source document ↗Export metadata ↓

Abstract

Multiphase compressible flows are often characterized by a broad range of space and time scales. Thus entailing large grids and small time steps, simulations of these flows on CPU-based clusters can thus take several wall-clock days. Offloading the compute kernels to GPUs appears attractive but is memory-bound for standard finite-volume and -difference methods, damping speed-ups. Even when realized, faster GPU-based kernels lead to more intrusive communication and I/O times. We present a portable strategy for GPU acceleration of multiphase compressible flow solvers that addresses these challenges and obtains large speedups at scale. We use OpenACC for portable offloading of all compute kernels while maintaining low-level control when needed. An established Fortran preprocessor and metaprogramming tool, Fypp, enables otherwise hidden compile-time optimizations. This strategy exposes compile-time optimizations and high memory reuse while retaining readable, maintainable, and compact code. Remote direct memory access, realized via CUDA-aware MPI, reduces communication times. We implement this approach in the open-source solver MFC. Metaprogramming-based preprocessing results in an 8-times speedup of the most expensive kernels, 46% of peak FLOPs on NVIDIA GPUs, and high arithmetic intensity (about 10 FLOPs/byte). In representative simulations, a single A100 GPU is 300-times faster than an Intel Xeon CPU core, corresponding to a 9-times speedup for a single A100 compared to the entire CPU die. At the same time, near-ideal (97%) weak scaling is observed for at least 13824 GPUs on Summit. A strong scaling efficiency of 84% is retained for an 8-times increase in GPU count. Collective I/O, implemented via MPI3, helps ensure negligible contribution of data transfers. Large many-GPU simulations of compressible (solid-)liquid-gas flows demonstrate the practical utility of this strategy.

Explore related subjects

physics.flu-dyn physics.comp-ph

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Anand Radhakrishnan, Henry Le Berre, Benjamin Wilfong, Jean-Sebastien Spratt, Mauro Rodriguez Jr., Tim Colonius, Spencer H. Bryngelson. 2024-02-03. Method for portable, scalable, and performant GPU-accelerated simulation of multiphase compressible flow. https://doi.org/10.1016/j.cpc.2024.109238

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Kinetic closure of turbulence

This letter presents a kinetic closure of the filtered Boltzmann--BGK equation, paving the way toward an alternative description of turbulence. The closure retains the turbulent subfilter stress tensor without a separate Smagorinsky-type ansatz for its structure, unlike classical filtered Navier--Stokes closures. In contrast, it accounts for the subfilter turbulent diffusion in the nonconserved moments by generalizing the BGK collision operator. The model does not require scale separation between resolved and unresolved scales. The Chapman--Enskog analysis shows how its hydrodynamic limit can converge to the filtered Navier--Stokes equations, with velocity gradients isolating subfilter contributions. Numerical tests on the Taylor--Green vortex and the turbulent mixing layer show improved stability and reduced dissipation in the reported cases, benchmarked against the Smagorinsky model.

physics.flu-dyn↗

arXiv2026

Kinetic closure of turbulence: collision-side modeling beyond the filtered Boltzmann equation

This article extends a recently introduced kinetic closure of turbulence by developing its theoretical framework, operational realizations, and validation. In contrast with filtered Navier--Stokes formulations, filtering the Boltzmann equation retains subgrid transport under the linear streaming operator, so that unresolved physics is concentrated on the collision side. We show that in dilute-gas large-eddy simulation, the main limitation of Bhatnagar--Gross--Krook (BGK)-type collision models is not the breakdown of molecular chaos, but the retention of a Markovian collision at a scale where filtering can induce finite temporal correlations in the collision product. In a BGK-type framework, the closure problem is dual: one must infer the filtered fine-grained equilibrium, not computable from filtered moments alone, and model the non-Markovian collision dynamics generated by the collision-product covariance. The present framework makes this dual structure explicit and represents the resulting collision-covariance source term through a BGK-like closure built from the equilibrium commutation residual, with the turbulent relaxation frequency given by a first phenomenological realization. The framework relies on a Chapman--Enskog analysis organized by the reference timescale ratio emerging from the nondimensionalization of the kinetic equation, performed in the classical sense, thereby avoiding artificial turbulent scale separations. We show that the Chapman--Enskog structure is not a pure one-parameter Knudsen scaling: the primary ordering is set by the kinetic-to-macroscopic timescale ratio, while higher moments retain an additional Mach dependence through the mixed scaling of particle velocity. The resulting kinetic closures are validated through lattice Boltzmann simulations and compared with the Smagorinsky model and regularization-based collision models.

physics.flu-dyn↗

arXiv2026

Deep Koopman Sensing

Real-time reconstruction of fluid flows from sparse sensor measurements is important for both physical understanding and flow control. When first-principles models are too expensive for online data assimilation (DA), learned reduced-order models provide an efficient alternative, but are commonly optimized for forward prediction rather than state estimation. We propose Deep Koopman Sensing, a data-driven reduced-order DA framework that combines a nonlinear autoencoder with parameter-conditioned linear latent dynamics approximating the Koopman operator. We compare the proposed model with parametric dynamic mode decomposition (pDMD), a multilayer perceptron (MLP), and xLSTM across four benchmarks: 1D viscous Burgers, 2D flow past a cylinder, 2D dambreak, and 3D flow past a sphere. Our results reveal a marked distinction between forecasting and sensing: open-loop accuracy does not reliably predict assimilation performance, while Deep Koopman Sensing achieves the lowest assimilation error across all four benchmarks. More importantly, with an extended Kalman filter, incorporating sensor measurements improves the estimates of both linear latent models across all four benchmarks, whereas it degrades the nonlinear models, despite their strong open-loop performance. With ensemble filtering, the nonlinear models are no longer degraded by assimilation, while the Koopman model still attains the lowest assimilation error. These results show that latent dynamics should be designed for the downstream estimation task rather than selected solely for forecast accuracy, and demonstrate Koopman-based reduced-order modeling as an effective approach for real-time flow reconstruction from sparse, streaming measurements.

physics.flu-dyn↗