On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication

Tadisetty Sai Yashwanth

Home/Archive/arXiv

arXiv · 2511.00025

On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication

Tadisetty Sai Yashwanth

View original source ↗Read source document ↗Export metadata ↓

Abstract

Floating-point non-associativity makes fundamental deep learning operations, such as matrix multiplication (matmul) on GPUs, inherently non-deterministic. Despite this, the statistical structure of the resulting numerical error remains poorly understood. A common working assumption is that these errors behave as independent and identically distributed (i.i.d.) Gaussian noise. In this paper, we empirically test this assumption and show that it fails to describe real GPU behavior. By comparing outputs of single-input and batched matmuls, we find that while the i.i.d. model predicts non-zero output instability, empirical results show a 0.00% prediction flip rate. Through covariance analysis, we uncover the cause: the floating-point error is structured and highly correlated. For float16, nearly 50% of the total error variance lies in off-diagonal terms, revealing that the noise behaves as a coordinated, directional perturbation rather than random static. This result challenges the prevailing stochastic view of numerical noise and provides a principled foundation for analyzing deep learning reliability under hardware non-determinism.

Explore related subjects

math.NA cs.LG cs.NA

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Tadisetty Sai Yashwanth. 2025-10-26. On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication. https://arxiv.org/abs/2511.00025

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Discrete normalized gradient flow for two-component Bose-Einstein condensates: Energy dissipation, global convergence and sharp local convergence behavior

The gradient flow with semi-implicit discretization (GFSI) is the most widely used algorithm for computing the ground state of Gross-Pitaevskii energy functional. We apply GFSI to the two-component scenario with Josephson junction and rotating term, which is one of the most important and topical models in multi-component Bose-Einstein condensates (MBECs), and rigorously establish the following fundamental results for the first time. By introducing a Lagrange multiplier to reformulate GFSI into an equivalent form, we prove its energy dissipation property and global convergence to stationary states. More significantly, we uncover an intrinsic connection between this classical numerical PDE discretization rooted in imaginary-time evolution and Riemannian optimization, a state-of-the-art mathematical framework for manifold-constrained optimization. This connection enables us to fully characterize the local convergence behavior of GFSI within the Riemannian optimization framework. Together with the aforementioned global convergence result, this yields a complete global--local convergence theory for GFSI. Finally, numerical experiments comprehensively validate the theoretically predicted energy dissipation and convergence properties.

math.NA↗

arXiv2026

Discretization, Uniform-in-Time Estimations and Approximation of Invariant Measures for Nonlinear Stochastic Differential Equations with Non-Uniform Dissipativity

The approximation of invariant measures for nonlinear ergodic stochastic differential equations (SDEs) is a central problem in scientific computing, with important applications in stochastic sampling, physics, and ecology. We first propose an easily applicable explicit Truncated Euler-Maruyama (TEM) scheme and prove its numerical ergodicity in the $L^p$-Wasserstein distance ($p\geqslant 1$). Furthermore, by combining truncation techniques with the coupling method, we establish a uniform-in-time $1/2$-order convergence rate in moments for the TEM scheme. Additionally, leveraging the exponential ergodicity of both the numerical and exact solutions, we derive a $1/2$-order convergence rate for the invariant measures of the TEM scheme and the exact solution in the $L^1$-Wasserstein distance. Finally, two numerical experiments are conducted to validate our theoretical results.

math.NA↗

arXiv2026

Barotropic-Baroclinic Splitting for Multilayer Shallow Water Models with Exchanges

This work presents the numerical analysis of a barotropic-baroclinic splitting in a nonlinear multilayer framework with exchanges between the layers in terrain-following coordinates. The splitting is formulated as an exact operator splitting. The barotropic step handles free surface evolution and depth-averaged velocity via a well-balanced one-layer model, while the baroclinic step manages vertical exchanges between layers and adjusts velocities to their mean values. We show that the barotropic-baroclinic splitting preserves total energy conservation and meets both a discrete maximum principle and a discrete entropy inequality. Several numerical experiments are presented showing the gain in computational cost, particularly in low Froude simulations, with no loss of accuracy. The benefits of using a well-balancing strategy in the barotropic step to preserve the geostrophic equilibrium are inherited in the overall scheme.

math.NA↗