Exact Convergence rate of the subgradient method by using Polyak step size

Abstract

This paper studies the last iterate of subgradient method with Polyak step size when applied to the minimization of a nonsmooth convex function with bounded subgradients. We show that the subgradient method with Polyak step size achieves a convergence rate $\mathcal{O}\left(\tfrac{1}{\sqrt[4]{N}}\right)$ in terms of the final iterate. An example is provided to show that this rate is exact and cannot be improved. We introduce an adaptive Polyak step size for which the subgradient method enjoys a convergence rate $\mathcal{O}\left(\tfrac{1}{\sqrt{N}}\right)$ for the last iterate. Its convergence rate matches exactly the lower bound on the performance of any black-box method on the considered problem class. Additionally, we propose an adaptive Polyak method with a momentum term, where the step sizes are independent of the number of iterates. We establish that the algorithm also attains the optimal convergence rate. We investigate the alternating projection method. We derive a convergence rate $\left( \frac{2N }{ 2N+1 } \right)^N\tfrac{R}{\sqrt{2N+1}}$ for the last iterate, where $R$ is a bound on the distance between the initial iterate and a solution. An example is also provided to illustrate the exactness of the rate.