Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians

Abstract

We propose a new first-order method for minimizing nonconvex functions with Lipschitz continuous gradients and Hölder continuous Hessians. The proposed algorithm is a heavy-ball method equipped with two particular restart mechanisms. It finds a solution where the gradient norm is less than $ε$ in $O(H_ν^{\frac{1}{2 + 2 ν}} ε^{- \frac{4 + 3 ν}{2 + 2 ν}})$ function and gradient evaluations, where $ν\in [0, 1]$ and $H_ν$ are the Hölder exponent and constant, respectively. Our algorithm is $ν$-independent and thus universal; it automatically achieves the above complexity bound with the optimal $ν\in [0, 1]$ without knowledge of $H_ν$. In addition, the algorithm does not require other problem-dependent parameters as input, including the gradient's Lipschitz constant or the target accuracy $ε$. Numerical results illustrate that the proposed method is promising.