A New Primal-Dual Algorithm with Convex Combination and Extrapolation for Convex-Concave Saddle Point Problems with Nonlinear Coupling Term

Abstract

Convex-concave saddle point problems with nonlinear coupling term have wide applications in signal processing, machine learning, robust optimization, and generative models, among others. Primal-dual algorithms are widely used for convex-concave saddle point problems. However, when handling nonlinear coupling term in convex-concave saddle point problems, primal-dual algorithms usually encounter geometric mismatches between variable and mapping spaces, delayed gradient information, and strong dependence on linesearch, resulting in less stable performance and complex convergence analysis. To address these issues, we propose a new primal-dual algorithm named PDAce by combining convex combination and extrapolation strategies. More specifically, in the update of the primal variable, we construct a convex combination point to replace the current iterative point and compute the Jacobian matrix of the vector function in the nonlinear coupling term at the convex combination point. Besides, we use the latest information of convex combination points to extend extrapolation to the mapping space in the update of the dual variable. The core innovations lie in performing linearization of the vector function in the nonlinear coupling term at convex combination points and shifting extrapolation from the variable space to the nonlinear mapping space. This design completely eliminates nonlinear residual terms and allows for rigorous convergence analysis without linesearch. Under mild convex assumptions, we construct a new Lyapunov potential function to prove that PDAce is globally convergent with an ergodic convergence rate of $\mathcal{O}(1/N)$. Moreover, we develop an accelerated version of PDAce, termed aPDAce, which achieves $\mathcal{O}(1/N^2)$ rate under strong convexity of the primal function, and linear convergence when both the primal and dual functions are strongly convex.