Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

Steffen Dereich; Thang Do; Arnulf Jentzen

Home/Archive/arXiv

arXiv · 2603.18899

Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

Steffen Dereich·Thang Do·Arnulf Jentzen

View original source ↗Read source document ↗Export metadata ↓

Abstract

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.

Explore related subjects

cs.LG math.OC

Keep this discovery

BibTeX RIS

Steffen Dereich, Thang Do, Arnulf Jentzen. 2026-09-01. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method. https://arxiv.org/abs/2603.18899

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.LG subject math.OC subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning

We prove a non-asymptotic central limit theorem for vector-valued martingale differences using Stein's method, and use Poisson's equation to extend the result to functions of Markov Chains. We then show that these results can be applied to establish a non-asymptotic central limit theorem for Temporal Difference (TD) learning with averaging.

math.PR↗

arXiv2025

Momentum-based gradient descent methods for Lie groups

Polyak's Heavy Ball (PHB; Polyak, 1964), a.k.a. Classical Momentum, and Nesterov's Accelerated Gradient (NAG; Nesterov, 1983) are well-established momentum-descent methods for optimization. Although the latter generally outperforms the former, primarily, generalizations of PHB-like methods to nonlinear spaces have not been sufficiently explored in the literature. In this paper, we propose a generalization of NAG-like methods for Lie group optimization. This generalization is based on the variational one-to-one correspondence between classical and accelerated momentum methods (Campos et al., 2023). We provide numerical experiments for chosen retractions on the group of rotations based on the Frobenius norm and the Rosenbrock function to demonstrate the effectiveness of our proposed methods, and that align with results of the Euclidean case, that is, a faster convergence rate for NAG.

math.OC↗

arXiv2026

Sample Complexity of Linear Quadratic Regulator Without Initial Stability

Inspired by REINFORCE, we introduce a novel receding-horizon algorithm for the Linear Quadratic Regulator (LQR) problem with unknown dynamics. Unlike prior methods, our algorithm avoids reliance on two-point gradient estimates while maintaining the same order of sample complexity. Furthermore, it eliminates the restrictive requirement of starting with a stable initial policy, broadening its applicability. Beyond these improvements, we introduce a refined analysis of error propagation through the contraction of the Riccati operator under the Riemannian distance. This refinement leads to a better sample complexity and ensures improved convergence guarantees.

math.OC↗