Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.32505

Fast Differentiable SVD on GPU via Polar Decomposition

Uliana Parkina·Askar Tsyganov·Sergei Kudriashov·Sergey Samsonov·Maxim Rakhuba
View original source ↗Read source document ↗Export metadata ↓

Abstract

We present a fully GPU-oriented SVD pipeline based on polar decomposition, motivated by iterative methods that rely solely on matrix multiplications, such as the Newton-Schulz iteration. We show that this approach enables up to a $2\times$ speedup compared to standard implementations. Furthermore, we derive a numerically stable backward pass for the polar decomposition and leverage it to obtain a fully differentiable SVD. Our methods are released as open-source implementations in both PyTorch and JAX: https://github.com/fallnlove/cans_svd.

Explore related subjects

math.NAcs.LGcs.NA

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Uliana Parkina, Askar Tsyganov, Sergei Kudriashov, Sergey Samsonov, Maxim Rakhuba. 2026-09-26. Fast Differentiable SVD on GPU via Polar Decomposition. https://arxiv.org/abs/2609.32505

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra

Natural Gradient Descent (NGD) has emerged as a promising optimization algorithm for training neural network-based solvers for partial differential equations (PDEs), such as Physics-Informed Neural Networks (PINNs). However, its practical use is often limited by the high computational cost of solving linear systems involving the Gramian matrix. While matrix-free NGD methods based on the conjugate gradient (CG) method avoid explicit matrix inversion, the ill-conditioning of the Gramian significantly slows the convergence of CG. In this work, we extend matrix-free NGD to broader classes of problems than previously considered and propose the use of Randomized Numerical Linear Algebra (RandNLA) techniques for efficient preconditioning of the inner CG solver. The resulting algorithms demonstrate substantial performance improvements over existing NGD-based methods on a range of PDE problems discretized using neural networks, and offer competitive results compared to other state-of-the-art optimizers.

math.NA↗
arXiv2026

Helmholtz boundary integral methods and the pollution effect

This paper is concerned with solving the Helmholtz exterior Dirichlet and Neumann problems with large wavenumber $k$ and smooth obstacles using the standard second-kind boundary integral equations. We consider Galerkin and collocation methods -- with subspaces consisting of $\textit{either}$ piecewise polynomials (in 2-d for collocation, in any dimension for Galerkin) $\textit{or}$ trigonometric polynomials (in 2-d) -- as well as a fully discrete quadrature (Nyström) method based on trigonometric polynomials (in 2-d). For each of these methods, we prove -- in many cases for the first time -- rigorous results about the fundamental question: how quickly must the number of degrees of freedom (the dimension of the approximation space) grow with $k$ to maintain accuracy of the computed solution? Importantly, we determine which of these methods suffer from $\textit{the pollution effect}$. That is, we address the question: must the number of points per wavelength $\to \infty$ to maintain accuracy as $k\to\infty$?

math.NA↗
arXiv2026

DE-Sinc approximation for unilateral rapidly decreasing functions and its computable error bound

The Sinc approximation is highly effective for functions that decay rapidly at both ends of the real axis. For unilateral rapidly decreasing functions, which decay algebraically as $t\to-\infty$ and exponentially as $t\to\infty$, an appropriate variable transformation is required. Existing single-exponential transformations for this class of functions yield only root-exponential convergence, even when improved transformations are employed. This paper develops a double-exponential (DE)-Sinc approximation based on the transformation $t=2\sinh(\log(\log(1+\exp(π\sinh x))))$, which was previously introduced for numerical integration. The main contribution is a rigorous and computable error bound of order $\operatorname{O}(\exp(-cn/\log n))$ with a constant explicitly expressed in terms of the problem parameters. Under the stated assumptions, the resulting approximation achieves almost exponential convergence and is suitable for computation with guaranteed accuracy. Numerical examples satisfying these assumptions confirm the predicted convergence behavior and the validity of the derived error bound.

math.NA↗