High-Probability Last-Iterate Guarantees for Two-Point Gaussian Zeroth-Order Stochastic Gradient Descent

Abstract

We establish a direct high-probability last-iterate guarantee for the standard same-sample two-point Gaussian zeroth-order SGD method in smooth, strongly convex stochastic optimization. At each iteration, the method draws a fresh Gaussian direction, evaluates two symmetric perturbations with the same stochastic sample, and takes a norm-normalized stochastic approximation step. Assuming unbiased stochastic gradients and a conditional exponential-moment bound on the squared norm of the stochastic gradient noise, we prove a finite-horizon bound, valid for dimension \(d\ge2\), with an explicit product-weight factor. When the offset in the stepsize schedule is large enough relative to the logarithmic confidence terms, this factor is bounded and the result gives \[ f(\bx_T)-f(\bx^*) = \widetilde{\mathcal O}\!\left(\frac{d}{T}\right) \] with probability at least \(1-δ\), up to fixed problem parameters and logarithmic factors. Thus the confidence dependence is logarithmic rather than polynomial in \(1/δ\), and the proof neither invokes Markov's inequality nor truncates the noise. To the best of our knowledge, this is the first direct high-probability last-iterate result at this zeroth-order scale for the same-sample Gaussian recursion under conditional sub-Gaussian stochastic-gradient noise. The proof combines uniform weighted lower and upper scans for Gaussian angles, a product-martingale boundary for the signed suffix-product term, and terminal nonnegative concentration estimates. We also formulate the resulting general pathwise framework for stochastic recursions with random contraction and signed perturbations, identifying the scan, filtration, variance, and terminal-control conditions under which the same argument applies.