arXiv · 2611.09596
Coordinatewise Epsilon for Adam under Heavy-Tailed Noise
Abstract
Adam estimates second moments coordinatewise, but typically adds the same scalar $\varepsilon$ to every denominator. We study how this shared $\varepsilon$ affects convergence when noise and curvature vary across coordinates. Under coordinatewise finite-$p$ noise moments, $p\in(1,2]$, choosing $\varepsilon_i$ from the noise and curvature bounds for each coordinate attains the heavy-tailed $\ell_1$-stationarity benchmark $O(T^{-\frac{p-1}{3p-2}})$, matching the known optimal time exponent for smooth nonconvex stochastic optimization. For $1<p<2$, Adam with a shared $\varepsilon_t$ admits a lower bound of order $Ω(T^{-\frac{p-1}{2p}})$. This lower bound holds for every predictable scalar choice of $η_t$, $β_{1,t}$, $β_{2,t}$, and $\varepsilon_t$, precluding a dimension-uniform benchmark rate for this class.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Jeonseong Kim, Chenglin Fan. 2026-11-06. Coordinatewise Epsilon for Adam under Heavy-Tailed Noise. https://arxiv.org/abs/2611.09596
Cite the original work for its findings. Save a collection to share your selection of sources.